Sep 17 15:36:50.097001 ip-10-0-128-144 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 15:36:50.097015 ip-10-0-128-144 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 15:36:50.097024 ip-10-0-128-144 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 15:36:50.097509 ip-10-0-128-144 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 15:36:59.711849 ip-10-0-128-144 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 17 15:36:59.711864 ip-10-0-128-144 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 571f56a3b8d64c69b96fd06740f3889c --
Sep 17 15:38:33.916198 ip-10-0-128-144 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:38:34.362792 ip-10-0-128-144 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:34.362792 ip-10-0-128-144 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:38:34.362792 ip-10-0-128-144 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:34.362792 ip-10-0-128-144 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:38:34.362792 ip-10-0-128-144 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:34.365348 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.365254    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:38:34.368457 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368436    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:34.368457 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368452    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:34.368457 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368458    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:34.368457 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368463    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368467    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368471    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368475    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368481    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368485    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368489    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368493    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368497    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368502    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368506    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368509    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368513    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368518    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368524    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368530    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368535    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368541    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368545    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368549    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:34.368689 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368553    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368558    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368562    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368566    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368570    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368574    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368578    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368582    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368587    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368591    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368595    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368599    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368603    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368608    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368613    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368617    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368621    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368626    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368630    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368634    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:34.369524 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368639    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368643    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368647    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368650    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368654    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368658    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368662    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368667    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368671    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368675    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368680    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368685    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368688    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368702    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368707    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368711    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368715    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368719    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368723    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:34.370219 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368727    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368731    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368735    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368739    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368743    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368747    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368752    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368757    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368762    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368767    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368771    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368775    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368779    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368786    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368791    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368795    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368800    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368804    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368810    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368814    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:34.370676 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368819    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368823    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368827    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368831    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368836    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368840    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368844    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368867    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368872    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368876    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368880    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368884    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368888    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368892    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368897    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368901    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.368907    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370050    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370062    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370066    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:34.371191 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370069    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370072    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370075    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370078    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370081    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370083    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370086    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370089    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370091    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370094    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370097    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370100    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370102    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370105    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370108    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370110    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370112    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370115    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370118    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370120    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370123    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:34.371685 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370125    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370128    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370130    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370132    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370135    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370138    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370140    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370142    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370145    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370148    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370151    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370154    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370157    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370160    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370162    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370165    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370167    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370170    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370172    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370175    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:34.372229 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370177    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370180    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370182    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370185    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370187    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370190    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370192    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370195    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370197    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370200    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370202    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370204    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370207    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370209    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370211    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370214    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370216    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370220    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370222    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:34.372722 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370225    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370230    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370235    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370238    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370241    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370244    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370247    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370249    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370252    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370254    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370257    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370260    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370262    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370265    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370267    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370269    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370275    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370278    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:34.373215 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370282    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370285    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370288    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370292    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370296    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370299    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370302    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370305    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370307    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370310    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370313    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370316    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370319    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370322    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370324    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370327    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370330    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370333    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370684    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370689    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:34.373663 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370692    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370694    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370697    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370701    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370705    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370708    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370712    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370716    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370718    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370722    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370725    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370727    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370730    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370732    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370735    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370737    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370739    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370742    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370745    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:34.374171 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370747    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370750    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370753    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370756    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370758    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370761    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370763    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370766    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370768    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370771    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370773    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370776    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370778    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370781    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370783    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370785    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370788    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370791    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370793    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370796    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:34.374639 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370798    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370801    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370803    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370805    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370808    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370811    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370813    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370815    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370818    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370820    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370823    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370825    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370827    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370830    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370833    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370837    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370840    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370842    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370845    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:34.375177 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370847    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370867    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370872    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370874    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370877    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370880    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370882    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370885    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370887    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370890    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370893    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370896    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370898    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370900    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370903    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370905    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370908    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370910    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370913    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370916    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:34.375646 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370919    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370921    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370924    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370926    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370928    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370931    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370934    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370940    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370942    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370945    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370947    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370950    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370952    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370954    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370957    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370960    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370962    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370964    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.370967    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371037    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371074    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:38:34.376156 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371081    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371085    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371090    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371093    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371097    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371101    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371104    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371107    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371111    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371114    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371117    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371120    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371123    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371126    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371129    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371132    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371136    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371139    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371144    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371147    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371150    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371154    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371157    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:38:34.376660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371160    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371163    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371166    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371169    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371172    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371175    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371178    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371182    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371185    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371187    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371190    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371194    2580 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371197    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371203    2580 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371207    2580 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371210    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371213    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371215    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371219    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371222    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371225    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371228    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371231    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371233    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371236    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:38:34.377235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371239    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371242    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371246    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371248    2580 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371252    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371255    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371258    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371261    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371264    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371267    2580 flags.go:64] FLAG: --help="false"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371270    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371273    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371276    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371279    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371282    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371285    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371288    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371291    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371294    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371297    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371299    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371302    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371305    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371308    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:38:34.377845 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371311    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371314    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371316    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371319    2580 flags.go:64] FLAG: --lock-file=""
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371323    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371326    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371329    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371334    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371336    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371339    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371343    2580 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371346    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371349    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371352    2580 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371355    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371359    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371362    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371366    2580 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371369    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371372    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371375    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371378    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371381    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371384    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371387    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:38:34.378579 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371404    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371407    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371410    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371413    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371416    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371421    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371423    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371426    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371430    2580 flags.go:64] FLAG: --port="10250"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371433    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371435    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a19d05f51f5de00a"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371438    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371442    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371445    2580 flags.go:64] FLAG: --register-node="true"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371448    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371451    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371454    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371457    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371461    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371465    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371468    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371470    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371473    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371476    2580 flags.go:64] FLAG: --runonce="false"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371479    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:38:34.379231 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371482    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371485    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371488    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371490    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371493    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371497    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371500    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371504    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371507    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371510    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371513    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371516    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371519    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371522    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371527    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371530    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371532    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371536    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371539    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371542    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371545    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371548    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371551    2580 flags.go:64] FLAG: --v="2"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371555    2580 flags.go:64] FLAG: --version="false"
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371559    2580 flags.go:64] FLAG: --vmodule=""
Sep 17 15:38:34.380129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371564    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.371567    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371680    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371684    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371687    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371690    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371694    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371698    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371701    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371704    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371706    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371709    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371712    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371714    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371718    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371721    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371723    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371726    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371730    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:34.380836 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371733    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371736    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371739    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371741    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371744    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371747    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371750    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371753    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371755    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371759    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371761    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371763    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371766    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371770    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371772    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371775    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371777    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371780    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371782    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371785    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:34.381310 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371787    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371790    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371792    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371795    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371799    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371802    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371805    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371808    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371811    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371815    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371817    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371820    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371822    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371824    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371827    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371829    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371832    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371834    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371837    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:34.381793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371839    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371842    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371845    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371847    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371867    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371870    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371875    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371877    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371880    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371883    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371886    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371889    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371891    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371894    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371896    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371899    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371902    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371904    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371907    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371909    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:34.382288 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371913    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371915    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371918    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371921    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371923    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371933    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371936    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371938    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371941    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371944    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371946    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371949    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371951    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371954    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371956    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371959    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371962    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371965    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371969    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371971    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371974    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:34.382810 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371976    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.371979    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.372657    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.381529    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.381546    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.381566    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.381572    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381627    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381631    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381635    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381638    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381641    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381644    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381647    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381650    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:34.383344 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381652    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381655    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381658    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381661    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381663    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381666    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381669    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381672    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381675    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381677    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381680    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381683    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381688    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381691    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381694    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381698    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381701    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381704    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381707    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:34.383728 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381716    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381721    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381724    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381727    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381730    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381733    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381736    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381739    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381741    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381744    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381746    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381749    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381752    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381754    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381757    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381759    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381762    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381764    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381767    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381770    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:34.384206 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381772    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381775    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381777    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381780    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381782    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381785    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381787    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381790    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381792    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381795    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381798    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381801    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381804    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381807    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381809    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381812    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381814    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381817    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381820    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381822    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:34.384700 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381825    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381827    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381830    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381833    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381835    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381838    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381840    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381843    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381846    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381848    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381866    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381868    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381871    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381874    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381877    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381880    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381882    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381886    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381890    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381893    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:34.385234 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381895    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381898    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381900    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381903    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381906    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381908    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381911    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381913    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381916    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381919    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381921    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.381924    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.381929    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382050    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:34.385730 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382054    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382057    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382060    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382062    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382065    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382067    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382070    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382072    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382075    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382078    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382080    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382085    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382088    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382090    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382093    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382095    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382099    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382102    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382105    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:34.386110 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382108    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382111    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382114    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382117    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382119    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382122    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382124    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382127    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382129    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382132    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382135    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382137    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382140    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382142    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382145    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382148    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382150    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382153    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382156    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:34.386598 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382160    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382163    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382166    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382169    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382171    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382173    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382177    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382179    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382182    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382185    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382187    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382190    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382192    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382194    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382197    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382200    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382202    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382205    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382207    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382211    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:34.387164 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382214    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382217    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382220    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382222    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382225    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382228    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382230    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382233    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382235    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382238    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382240    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382242    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382245    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382247    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382250    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382252    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382255    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382257    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382260    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:34.387673 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382264    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382266    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382269    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382271    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382274    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382276    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382279    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382281    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382284    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382286    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382289    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382292    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382295    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382297    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382300    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382302    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382305    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382308    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382310    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382312    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:34.388277 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:34.382315    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:34.388759 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.382319    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:34.388759 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.382421    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:38:34.388759 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.385911    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:38:34.388759 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.386906    2580 server.go:1019] "Starting client certificate rotation"
Sep 17 15:38:34.388759 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.387013    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:38:34.388759 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.387052    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:38:34.389652 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.389638    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:38:34.393394 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.393377    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:38:34.405325 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.405308    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:38:34.408505 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.408491    2580 log.go:25] "Validated CRI v1 image API"
Sep 17 15:38:34.409385 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.409368    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:38:34.413434 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.413417    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:38:34.416391 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.416374    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 de272a4c-847d-4a4a-8413-6b67aad14c35:/dev/nvme0n1p4 f8e97db8-fe68-44ac-a9f6-e283d9d92041:/dev/nvme0n1p3]
Sep 17 15:38:34.416445 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.416391    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:38:34.423226 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.423107    2580 manager.go:216] Machine: {Timestamp:2026-09-17 15:38:34.422565998 +0000 UTC m=+0.380875296 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112734 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f3a352985b6b26c3143372cb753de SystemUUID:ec2f3a35-2985-b6b2-6c31-43372cb753de BootID:571f56a3-b8d6-4c69-b96f-d06740f3889c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:26:ea:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:26:ea:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:71:70:35:60:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:38:34.423226 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.423219    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:38:34.423350 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.423334    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:38:34.425750 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.425721    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:38:34.425901 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.425754    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-144.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:38:34.426985 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.426976    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:38:34.427023 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.426989    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:38:34.427023 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.426997    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:38:34.427683 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.427672    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:38:34.428568 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.428559    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:38:34.432564 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.432552    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:38:34.432687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.432677    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:38:34.432716 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.432709    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:38:34.432743 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.432724    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:38:34.432770 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.432743    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:38:34.432770 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.432756    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:38:34.432770 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.432768    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:38:34.433184 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.433171    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:38:34.433255 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.433196    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:38:34.436237 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.436220    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 15:38:34.436495 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.436458    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8sps7"
Sep 17 15:38:34.438506 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.438489    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:38:34.438592 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.438528    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:38:34.440121 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440106    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:38:34.440198 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440140    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:38:34.440198 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440152    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:38:34.440198 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440161    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:38:34.440198 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440176    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:38:34.440198 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440194    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:38:34.440399 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440215    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:38:34.440399 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440226    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:38:34.440399 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440238    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:38:34.440399 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440253    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:38:34.440399 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440266    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:38:34.440399 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.440275    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:38:34.441108 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.441098    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:38:34.441165 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.441110    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:38:34.442183 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.442155    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:38:34.442260 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.442184    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:38:34.443612 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.443595    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-144.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:38:34.444900 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.444885    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8sps7"
Sep 17 15:38:34.445936 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.445923    2580 server.go:1268] "Started kubelet"
Sep 17 15:38:34.446105 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.446010    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:38:34.446159 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.446147    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:38:34.446485 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.446456    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:38:34.446734 ip-10-0-128-144 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:38:34.447623 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.447596    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:38:34.448951 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.448934    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:38:34.452244 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.452218    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:38:34.453240 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.452238    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-144.ec2.internal.18d625d8ab721442  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-144.ec2.internal,UID:ip-10-0-128-144.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-144.ec2.internal,},FirstTimestamp:2026-09-17 15:38:34.445894722 +0000 UTC m=+0.404204021,LastTimestamp:2026-09-17 15:38:34.445894722 +0000 UTC m=+0.404204021,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-144.ec2.internal,}"
Sep 17 15:38:34.453681 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.453666    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:38:34.454350 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.454331    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:38:34.454526 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.454500    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:34.454994 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.454976    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:38:34.455084 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.454998    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:38:34.455084 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.454978    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:38:34.455199 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.455101    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:38:34.455199 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.455106    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:38:34.456302 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.456279    2580 factory.go:55] Registering systemd factory
Sep 17 15:38:34.456397 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.456309    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:38:34.456623 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.456607    2580 factory.go:153] Registering CRI-O factory
Sep 17 15:38:34.456623 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.456621    2580 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:38:34.456735 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.456666    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:38:34.456783 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.456765    2580 factory.go:103] Registering Raw factory
Sep 17 15:38:34.456783 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.456778    2580 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:38:34.457186 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.457173    2580 manager.go:349] Starting recovery of all containers
Sep 17 15:38:34.465351 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.465323    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:34.469062 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.469041    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-144.ec2.internal\" not found" node="ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.469645 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.469628    2580 manager.go:354] Recovery completed
Sep 17 15:38:34.473574 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.473562    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:34.478717 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.478699    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:34.478793 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.478730    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:34.478793 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.478743    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:34.479399 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.479376    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:38:34.479399 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.479398    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:38:34.479530 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.479404    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:34.479530 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.479416    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:38:34.480159 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.480142    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:34.480235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.480171    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:34.480235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.480185    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:34.481574 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.481559    2580 policy_none.go:49] "None policy: Start"
Sep 17 15:38:34.481574 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.481576    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:38:34.481665 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.481585    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:38:34.483605 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.483594    2580 policy_none.go:47] "Start"
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.525385    2580 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.525600    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.525614    2580 server.go:87] "Starting device plugin registration server"
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.525919    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.525930    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.526025    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.526095    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.526103    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.526736    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:38:34.534075 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.526776    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:34.551517 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.551478    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:38:34.552842 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.552826    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:38:34.552927 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.552903    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:38:34.552976 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.552931    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:38:34.553043 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.553028    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:38:34.555402 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.555374    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:34.626761 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.626652    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:34.627743 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.627727    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:34.627827 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.627757    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:34.627827 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.627767    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:34.627827 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.627789    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.636764 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.636745    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.636834 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.636766    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-144.ec2.internal\": node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:34.653843 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.653817    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"]
Sep 17 15:38:34.653971 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.653900    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:34.655103 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.655086    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:34.655192 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.655119    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:34.655192 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.655130    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:34.656418 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.656402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.656483 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.656426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.657333 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.657320    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:34.657482 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.657469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.657518 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.657496    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:34.658120 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.658104    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:34.658185 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.658104    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:34.658185 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.658135    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:34.658185 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.658145    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:34.658185 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.658155    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:34.658185 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.658168    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:34.659048 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.659029    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:34.660089 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.660075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.660138 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.660100    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:34.660799 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.660783    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:34.660885 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.660813    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:34.660885 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.660828    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:34.683639 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.683611    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-144.ec2.internal\" not found" node="ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.687978 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.687963    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-144.ec2.internal\" not found" node="ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.756981 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.756951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.757082 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.756983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37310a8518ae4b4c8f438099a305077e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-144.ec2.internal\" (UID: \"37310a8518ae4b4c8f438099a305077e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.757082 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.757002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.757082 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.757052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.757191 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.757104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.759479 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.759462    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:34.857398 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.857351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37310a8518ae4b4c8f438099a305077e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-144.ec2.internal\" (UID: \"37310a8518ae4b4c8f438099a305077e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.857398 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.857409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37310a8518ae4b4c8f438099a305077e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-144.ec2.internal\" (UID: \"37310a8518ae4b4c8f438099a305077e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.860456 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.860436    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:34.961293 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:34.961233    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:34.986395 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.986377    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:34.991170 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:34.991027    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:35.062040 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.061998    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:35.162509 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.162469    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:35.263133 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.263059    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:35.363735 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.363702    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 17 15:38:35.386904 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.386876    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:38:35.387035 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.387012    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:38:35.387075 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.387013    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:38:35.407736 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.407711    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:35.433392 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.433365    2580 apiserver.go:52] "Watching apiserver"
Sep 17 15:38:35.446029 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.446008    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:38:35.446315 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.446295    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl","openshift-dns/node-resolver-hv6gp","openshift-image-registry/node-ca-rwm2w","openshift-multus/multus-5fm5g","openshift-network-diagnostics/network-check-target-jw9rz","openshift-network-operator/iptables-alerter-nrj2z","kube-system/global-pull-secret-syncer-j755r","kube-system/konnectivity-agent-88mrl","openshift-cluster-node-tuning-operator/tuned-d8v86","openshift-multus/multus-additional-cni-plugins-8t2t7","openshift-multus/network-metrics-daemon-xztjr","openshift-ovn-kubernetes/ovnkube-node-zpzbl"]
Sep 17 15:38:35.448910 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.448887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.450349 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.450307    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:33:35 +0000 UTC" deadline="2028-06-21 04:02:11.27280586 +0000 UTC"
Sep 17 15:38:35.450421 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.450347    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15420h23m35.822461186s"
Sep 17 15:38:35.451030 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.450992    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:38:35.451113 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.451077    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g5gmb\""
Sep 17 15:38:35.451113 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.451092    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:38:35.451113 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.451100    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:38:35.451113 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.451104    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.452825 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.452798    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:38:35.452932 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.452870    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:38:35.452989 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.452932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nkghk\""
Sep 17 15:38:35.453275 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.453260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.454005 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.453986    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:38:35.454509 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.454490    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:35.455100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.455083    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:38:35.455198 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.455081    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:38:35.455260 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.455198    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:38:35.455385 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.455367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.455464 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.455421    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-npq57\""
Sep 17 15:38:35.457148 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.457117    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:38:35.457287 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.457271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vjv9q\""
Sep 17 15:38:35.457362 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.457351    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:38:35.457417 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.457377    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:38:35.457469 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.457448    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:38:35.457625 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.457591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:35.457704 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.457682    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:35.459867 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.459831    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.460984 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.460966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcndj\" (UniqueName: \"kubernetes.io/projected/6ecaf429-99cd-48a9-8eaf-96cdc04dd17b-kube-api-access-tcndj\") pod \"node-resolver-hv6gp\" (UID: \"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b\") " pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.461074 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.460990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62b44d6a-61ff-458c-917c-0f4a105a381d-serviceca\") pod \"node-ca-rwm2w\" (UID: \"62b44d6a-61ff-458c-917c-0f4a105a381d\") " pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.461074 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-var-lib-kubelet\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461074 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-conf-dir\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461074 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-etc-kubernetes\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461074 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-socket-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.461285 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-registration-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.461285 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-device-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.461285 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpwp\" (UniqueName: \"kubernetes.io/projected/62b44d6a-61ff-458c-917c-0f4a105a381d-kube-api-access-7rpwp\") pod \"node-ca-rwm2w\" (UID: \"62b44d6a-61ff-458c-917c-0f4a105a381d\") " pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.461285 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-cni-binary-copy\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461285 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-var-lib-cni-bin\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461285 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-daemon-config\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461285 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ecaf429-99cd-48a9-8eaf-96cdc04dd17b-hosts-file\") pod \"node-resolver-hv6gp\" (UID: \"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b\") " pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.461594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-cni-dir\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-var-lib-cni-multus\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62b44d6a-61ff-458c-917c-0f4a105a381d-host\") pod \"node-ca-rwm2w\" (UID: \"62b44d6a-61ff-458c-917c-0f4a105a381d\") " pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.461594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-run-multus-certs\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-etc-selinux\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.461594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-hostroot\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bw2\" (UniqueName: \"kubernetes.io/projected/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-kube-api-access-z7bw2\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.461594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-sys-fs\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.461594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecaf429-99cd-48a9-8eaf-96cdc04dd17b-tmp-dir\") pod \"node-resolver-hv6gp\" (UID: \"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b\") " pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-socket-dir-parent\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461658    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4768\" (UniqueName: \"kubernetes.io/projected/9ff7e04e-bc8e-47db-aa47-09030f87b97a-kube-api-access-m4768\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-cnibin\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-run-k8s-cni-cncf-io\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-system-cni-dir\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-run-netns\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461878    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:38:35.462007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.461952    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w6tpk\""
Sep 17 15:38:35.462388 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.462006    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:38:35.462388 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.462066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:35.462388 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.462127    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:35.464252 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.464231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:38:35.464872 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.464628    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:38:35.464872 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.464715    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 17 15:38:35.466105 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.466083    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:38:35.466195 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.466179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:38:35.466368 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.466356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7krrp\""
Sep 17 15:38:35.466559 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.466546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.466939 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.466925    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:38:35.468235 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.468220    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f95f2\""
Sep 17 15:38:35.468404 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.468389    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:38:35.468520 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.468505    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:38:35.468734 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.468721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.470468 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.470452    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mrdhg\""
Sep 17 15:38:35.470554 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.470487    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:38:35.470724 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.470711    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:38:35.471001 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.470989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:35.471102 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.471078    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:35.475730 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.473650    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"]
Sep 17 15:38:35.475730 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.473932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.475730 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.475237    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:38:35.475730 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.475546    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"]
Sep 17 15:38:35.476375 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.476353    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:38:35.476480 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.476463    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:38:35.476543 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.476503    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:38:35.476623 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.476605    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:38:35.476749 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.476734    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:38:35.476809 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.476750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2qkg\""
Sep 17 15:38:35.476944 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.476887    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:38:35.484949 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.484931    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tps5p"
Sep 17 15:38:35.493577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.493552    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tps5p"
Sep 17 15:38:35.556631 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.556603    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:38:35.562115 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-tuned\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.562244 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-etc-selinux\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.562244 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-hostroot\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.562244 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-sysctl-conf\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.562244 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7c82c64-126e-4dce-9f44-a1279a7de1e4-konnectivity-ca\") pod \"konnectivity-agent-88mrl\" (UID: \"a7c82c64-126e-4dce-9f44-a1279a7de1e4\") " pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:38:35.562448 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-etc-selinux\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.562448 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-hostroot\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.562448 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-cni-netd\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.562448 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecaf429-99cd-48a9-8eaf-96cdc04dd17b-tmp-dir\") pod \"node-resolver-hv6gp\" (UID: \"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b\") " pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.562448 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-socket-dir-parent\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0814fe57-0b03-485c-bd00-544675e5ec2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0814fe57-0b03-485c-bd00-544675e5ec2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-socket-dir-parent\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4768\" (UniqueName: \"kubernetes.io/projected/9ff7e04e-bc8e-47db-aa47-09030f87b97a-kube-api-access-m4768\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-cnibin\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-run-k8s-cni-cncf-io\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-sys\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29378f5b-d5cc-4654-ac7c-f0baf5c85832-tmp\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0814fe57-0b03-485c-bd00-544675e5ec2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecaf429-99cd-48a9-8eaf-96cdc04dd17b-tmp-dir\") pod \"node-resolver-hv6gp\" (UID: \"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b\") " pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-ovnkube-script-lib\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-var-lib-kubelet\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-slash\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.562782 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-cnibin\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgj2z\" (UniqueName: \"kubernetes.io/projected/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-kube-api-access-lgj2z\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-run-k8s-cni-cncf-io\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62b44d6a-61ff-458c-917c-0f4a105a381d-serviceca\") pod \"node-ca-rwm2w\" (UID: \"62b44d6a-61ff-458c-917c-0f4a105a381d\") " pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-var-lib-kubelet\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7c82c64-126e-4dce-9f44-a1279a7de1e4-agent-certs\") pod \"konnectivity-agent-88mrl\" (UID: \"a7c82c64-126e-4dce-9f44-a1279a7de1e4\") " pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.562956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-log-socket\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-var-lib-kubelet\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-cni-binary-copy\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-var-lib-cni-bin\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-daemon-config\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ce38b5e-309f-42e2-a80b-43188cbbbf93-iptables-alerter-script\") pod \"iptables-alerter-nrj2z\" (UID: \"4ce38b5e-309f-42e2-a80b-43188cbbbf93\") " pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-sysctl-d\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-systemd-units\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ecaf429-99cd-48a9-8eaf-96cdc04dd17b-hosts-file\") pod \"node-resolver-hv6gp\" (UID: \"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b\") " pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-cni-dir\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.563511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62b44d6a-61ff-458c-917c-0f4a105a381d-serviceca\") pod \"node-ca-rwm2w\" (UID: \"62b44d6a-61ff-458c-917c-0f4a105a381d\") " pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-var-lib-cni-multus\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ecaf429-99cd-48a9-8eaf-96cdc04dd17b-hosts-file\") pod \"node-resolver-hv6gp\" (UID: \"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b\") " pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4624302-fc21-4d0e-8c9a-fdfd494389b3-kubelet-config\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-run-netns\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-cni-dir\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-var-lib-cni-multus\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-env-overrides\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-var-lib-cni-bin\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62b44d6a-61ff-458c-917c-0f4a105a381d-host\") pod \"node-ca-rwm2w\" (UID: \"62b44d6a-61ff-458c-917c-0f4a105a381d\") " pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-kubernetes\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62b44d6a-61ff-458c-917c-0f4a105a381d-host\") pod \"node-ca-rwm2w\" (UID: \"62b44d6a-61ff-458c-917c-0f4a105a381d\") " pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-run-openvswitch\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-run-ovn\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bw2\" (UniqueName: \"kubernetes.io/projected/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-kube-api-access-z7bw2\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:35.564099 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-cni-binary-copy\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-daemon-config\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-etc-openvswitch\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-cni-bin\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-ovnkube-config\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-ovn-node-metrics-cert\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-sys-fs\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-var-lib-openvswitch\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-sysconfig\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmjfm\" (UniqueName: \"kubernetes.io/projected/29378f5b-d5cc-4654-ac7c-f0baf5c85832-kube-api-access-fmjfm\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0814fe57-0b03-485c-bd00-544675e5ec2c-system-cni-dir\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0814fe57-0b03-485c-bd00-544675e5ec2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-sys-fs\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-run-systemd\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-system-cni-dir\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.564641 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-run-netns\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.563985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866dq\" (UniqueName: \"kubernetes.io/projected/4ce38b5e-309f-42e2-a80b-43188cbbbf93-kube-api-access-866dq\") pod \"iptables-alerter-nrj2z\" (UID: \"4ce38b5e-309f-42e2-a80b-43188cbbbf93\") " pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-systemd\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-kubelet\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-system-cni-dir\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcndj\" (UniqueName: \"kubernetes.io/projected/6ecaf429-99cd-48a9-8eaf-96cdc04dd17b-kube-api-access-tcndj\") pod \"node-resolver-hv6gp\" (UID: \"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b\") " pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-run-netns\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-conf-dir\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-etc-kubernetes\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-socket-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-registration-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-etc-kubernetes\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-multus-conf-dir\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-device-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-registration-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpwp\" (UniqueName: \"kubernetes.io/projected/62b44d6a-61ff-458c-917c-0f4a105a381d-kube-api-access-7rpwp\") pod \"node-ca-rwm2w\" (UID: \"62b44d6a-61ff-458c-917c-0f4a105a381d\") " pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-device-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.565100 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ff7e04e-bc8e-47db-aa47-09030f87b97a-socket-dir\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4624302-fc21-4d0e-8c9a-fdfd494389b3-dbus\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-host\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564294    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskgj\" (UniqueName: \"kubernetes.io/projected/0814fe57-0b03-485c-bd00-544675e5ec2c-kube-api-access-bskgj\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx47z\" (UniqueName: \"kubernetes.io/projected/d86c0b23-dea0-497e-a3ca-fa1037439372-kube-api-access-mx47z\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ce38b5e-309f-42e2-a80b-43188cbbbf93-host-slash\") pod \"iptables-alerter-nrj2z\" (UID: \"4ce38b5e-309f-42e2-a80b-43188cbbbf93\") " pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-run\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-lib-modules\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0814fe57-0b03-485c-bd00-544675e5ec2c-cnibin\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-node-log\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-run-multus-certs\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-modprobe-d\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.565597 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.564582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-host-run-multus-certs\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.570339 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.570316    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:38:35.574017 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.573986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bw2\" (UniqueName: \"kubernetes.io/projected/8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2-kube-api-access-z7bw2\") pod \"multus-5fm5g\" (UID: \"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2\") " pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.574132 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.574048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpwp\" (UniqueName: \"kubernetes.io/projected/62b44d6a-61ff-458c-917c-0f4a105a381d-kube-api-access-7rpwp\") pod \"node-ca-rwm2w\" (UID: \"62b44d6a-61ff-458c-917c-0f4a105a381d\") " pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.574132 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.574060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4768\" (UniqueName: \"kubernetes.io/projected/9ff7e04e-bc8e-47db-aa47-09030f87b97a-kube-api-access-m4768\") pod \"aws-ebs-csi-driver-node-5svpl\" (UID: \"9ff7e04e-bc8e-47db-aa47-09030f87b97a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.574132 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.574089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcndj\" (UniqueName: \"kubernetes.io/projected/6ecaf429-99cd-48a9-8eaf-96cdc04dd17b-kube-api-access-tcndj\") pod \"node-resolver-hv6gp\" (UID: \"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b\") " pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.606489 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:35.606461    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37310a8518ae4b4c8f438099a305077e.slice/crio-cc1de874528080a4521b81b3161de59e35e18bf4d4e785d1f3863d2933fa6bf2 WatchSource:0}: Error finding container cc1de874528080a4521b81b3161de59e35e18bf4d4e785d1f3863d2933fa6bf2: Status 404 returned error can't find the container with id cc1de874528080a4521b81b3161de59e35e18bf4d4e785d1f3863d2933fa6bf2
Sep 17 15:38:35.606727 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:35.606707    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a022f343c02e056acab4e9c92407e8c.slice/crio-8ed5fa4dbd4391da69039813dee30527d011e157947cdfaa5b891fbcba9a7ad6 WatchSource:0}: Error finding container 8ed5fa4dbd4391da69039813dee30527d011e157947cdfaa5b891fbcba9a7ad6: Status 404 returned error can't find the container with id 8ed5fa4dbd4391da69039813dee30527d011e157947cdfaa5b891fbcba9a7ad6
Sep 17 15:38:35.611695 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.611675    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:38:35.665360 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:35.665360 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-etc-openvswitch\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.665557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-cni-bin\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.665557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-ovnkube-config\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.665557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-ovn-node-metrics-cert\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.665557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-etc-openvswitch\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.665557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-cni-bin\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.665557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-var-lib-openvswitch\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.665557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-sysconfig\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.665557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-var-lib-openvswitch\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjfm\" (UniqueName: \"kubernetes.io/projected/29378f5b-d5cc-4654-ac7c-f0baf5c85832-kube-api-access-fmjfm\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0814fe57-0b03-485c-bd00-544675e5ec2c-system-cni-dir\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0814fe57-0b03-485c-bd00-544675e5ec2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-run-systemd\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-sysconfig\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-run-systemd\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866dq\" (UniqueName: \"kubernetes.io/projected/4ce38b5e-309f-42e2-a80b-43188cbbbf93-kube-api-access-866dq\") pod \"iptables-alerter-nrj2z\" (UID: \"4ce38b5e-309f-42e2-a80b-43188cbbbf93\") " pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0814fe57-0b03-485c-bd00-544675e5ec2c-system-cni-dir\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-systemd\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-kubelet\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4624302-fc21-4d0e-8c9a-fdfd494389b3-dbus\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-host\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-systemd\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskgj\" (UniqueName: \"kubernetes.io/projected/0814fe57-0b03-485c-bd00-544675e5ec2c-kube-api-access-bskgj\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.666028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-kubelet\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4624302-fc21-4d0e-8c9a-fdfd494389b3-dbus\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.665955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx47z\" (UniqueName: \"kubernetes.io/projected/d86c0b23-dea0-497e-a3ca-fa1037439372-kube-api-access-mx47z\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ce38b5e-309f-42e2-a80b-43188cbbbf93-host-slash\") pod \"iptables-alerter-nrj2z\" (UID: \"4ce38b5e-309f-42e2-a80b-43188cbbbf93\") " pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-host\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-run\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ce38b5e-309f-42e2-a80b-43188cbbbf93-host-slash\") pod \"iptables-alerter-nrj2z\" (UID: \"4ce38b5e-309f-42e2-a80b-43188cbbbf93\") " pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-lib-modules\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-run\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0814fe57-0b03-485c-bd00-544675e5ec2c-cnibin\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-ovnkube-config\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-node-log\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0814fe57-0b03-485c-bd00-544675e5ec2c-cnibin\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0814fe57-0b03-485c-bd00-544675e5ec2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-node-log\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-modprobe-d\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.666847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-tuned\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666219    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-lib-modules\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-sysctl-conf\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7c82c64-126e-4dce-9f44-a1279a7de1e4-konnectivity-ca\") pod \"konnectivity-agent-88mrl\" (UID: \"a7c82c64-126e-4dce-9f44-a1279a7de1e4\") " pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-cni-netd\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-modprobe-d\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0814fe57-0b03-485c-bd00-544675e5ec2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0814fe57-0b03-485c-bd00-544675e5ec2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-sys\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29378f5b-d5cc-4654-ac7c-f0baf5c85832-tmp\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0814fe57-0b03-485c-bd00-544675e5ec2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-sysctl-conf\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-ovnkube-script-lib\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-var-lib-kubelet\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-cni-netd\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-slash\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgj2z\" (UniqueName: \"kubernetes.io/projected/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-kube-api-access-lgj2z\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7c82c64-126e-4dce-9f44-a1279a7de1e4-agent-certs\") pod \"konnectivity-agent-88mrl\" (UID: \"a7c82c64-126e-4dce-9f44-a1279a7de1e4\") " pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-log-socket\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0814fe57-0b03-485c-bd00-544675e5ec2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ce38b5e-309f-42e2-a80b-43188cbbbf93-iptables-alerter-script\") pod \"iptables-alerter-nrj2z\" (UID: \"4ce38b5e-309f-42e2-a80b-43188cbbbf93\") " pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-sysctl-d\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-systemd-units\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4624302-fc21-4d0e-8c9a-fdfd494389b3-kubelet-config\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-run-netns\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-env-overrides\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-kubernetes\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7c82c64-126e-4dce-9f44-a1279a7de1e4-konnectivity-ca\") pod \"konnectivity-agent-88mrl\" (UID: \"a7c82c64-126e-4dce-9f44-a1279a7de1e4\") " pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-run-openvswitch\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-run-ovn\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.666941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-run-ovn\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.668835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0814fe57-0b03-485c-bd00-544675e5ec2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4624302-fc21-4d0e-8c9a-fdfd494389b3-kubelet-config\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-slash\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-log-socket\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-var-lib-kubelet\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-ovnkube-script-lib\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.667386    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-kubernetes\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.667480    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret podName:b4624302-fc21-4d0e-8c9a-fdfd494389b3 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:36.167442268 +0000 UTC m=+2.125751570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret") pod "global-pull-secret-syncer-j755r" (UID: "b4624302-fc21-4d0e-8c9a-fdfd494389b3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-systemd-units\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-host-run-netns\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-sys\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0814fe57-0b03-485c-bd00-544675e5ec2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ce38b5e-309f-42e2-a80b-43188cbbbf93-iptables-alerter-script\") pod \"iptables-alerter-nrj2z\" (UID: \"4ce38b5e-309f-42e2-a80b-43188cbbbf93\") " pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.667761    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.667807    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs podName:d86c0b23-dea0-497e-a3ca-fa1037439372 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:36.167789602 +0000 UTC m=+2.126098890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs") pod "network-metrics-daemon-xztjr" (UID: "d86c0b23-dea0-497e-a3ca-fa1037439372") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-run-openvswitch\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.669687 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.667891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-sysctl-d\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.670569 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.668123    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-env-overrides\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.670569 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.668356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-ovn-node-metrics-cert\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.670569 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.668645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29378f5b-d5cc-4654-ac7c-f0baf5c85832-etc-tuned\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.670569 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.669276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29378f5b-d5cc-4654-ac7c-f0baf5c85832-tmp\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.670569 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.669767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7c82c64-126e-4dce-9f44-a1279a7de1e4-agent-certs\") pod \"konnectivity-agent-88mrl\" (UID: \"a7c82c64-126e-4dce-9f44-a1279a7de1e4\") " pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:38:35.673901 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.673879    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:35.673901 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.673901    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:35.674097 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.673915    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s7jmc for pod openshift-network-diagnostics/network-check-target-jw9rz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:35.674097 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:35.673972    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc podName:da28f9b8-8cc1-45d7-a6b7-aebff23cc910 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:36.173957677 +0000 UTC m=+2.132266980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s7jmc" (UniqueName: "kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc") pod "network-check-target-jw9rz" (UID: "da28f9b8-8cc1-45d7-a6b7-aebff23cc910") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:35.675410 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.675326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjfm\" (UniqueName: \"kubernetes.io/projected/29378f5b-d5cc-4654-ac7c-f0baf5c85832-kube-api-access-fmjfm\") pod \"tuned-d8v86\" (UID: \"29378f5b-d5cc-4654-ac7c-f0baf5c85832\") " pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:35.675937 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.675916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-866dq\" (UniqueName: \"kubernetes.io/projected/4ce38b5e-309f-42e2-a80b-43188cbbbf93-kube-api-access-866dq\") pod \"iptables-alerter-nrj2z\" (UID: \"4ce38b5e-309f-42e2-a80b-43188cbbbf93\") " pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.676028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.676005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskgj\" (UniqueName: \"kubernetes.io/projected/0814fe57-0b03-485c-bd00-544675e5ec2c-kube-api-access-bskgj\") pod \"multus-additional-cni-plugins-8t2t7\" (UID: \"0814fe57-0b03-485c-bd00-544675e5ec2c\") " pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:35.676356 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.676337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx47z\" (UniqueName: \"kubernetes.io/projected/d86c0b23-dea0-497e-a3ca-fa1037439372-kube-api-access-mx47z\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:35.676992 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.676968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgj2z\" (UniqueName: \"kubernetes.io/projected/ce14c586-76d1-4bf1-8f04-f261b9afc9ce-kube-api-access-lgj2z\") pod \"ovnkube-node-zpzbl\" (UID: \"ce14c586-76d1-4bf1-8f04-f261b9afc9ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:35.778479 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.778448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl"
Sep 17 15:38:35.788278 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:35.788247    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff7e04e_bc8e_47db_aa47_09030f87b97a.slice/crio-854cda563396481dc986e56ff2c3c20c31b7d1c99c7c26d5ec9a054514cb2487 WatchSource:0}: Error finding container 854cda563396481dc986e56ff2c3c20c31b7d1c99c7c26d5ec9a054514cb2487: Status 404 returned error can't find the container with id 854cda563396481dc986e56ff2c3c20c31b7d1c99c7c26d5ec9a054514cb2487
Sep 17 15:38:35.791314 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.791297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hv6gp"
Sep 17 15:38:35.797707 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.797684    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rwm2w"
Sep 17 15:38:35.799240 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:35.799219    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecaf429_99cd_48a9_8eaf_96cdc04dd17b.slice/crio-147e58cf6cee11c1abce8a611a7e4dd4af91ce172efd003fbf3f54a7bbdf0bbe WatchSource:0}: Error finding container 147e58cf6cee11c1abce8a611a7e4dd4af91ce172efd003fbf3f54a7bbdf0bbe: Status 404 returned error can't find the container with id 147e58cf6cee11c1abce8a611a7e4dd4af91ce172efd003fbf3f54a7bbdf0bbe
Sep 17 15:38:35.805938 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:35.805914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b44d6a_61ff_458c_917c_0f4a105a381d.slice/crio-55861f0c966f822007a0c83ba74053a528883ab28312355b1765fa6ce34a5d53 WatchSource:0}: Error finding container 55861f0c966f822007a0c83ba74053a528883ab28312355b1765fa6ce34a5d53: Status 404 returned error can't find the container with id 55861f0c966f822007a0c83ba74053a528883ab28312355b1765fa6ce34a5d53
Sep 17 15:38:35.812247 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.812224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5fm5g"
Sep 17 15:38:35.820313 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:35.820293    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9cd823_b4d5_4fb8_8a9f_4c4f28a170c2.slice/crio-d46771b77e73219b39ac090682c1a4e8f2bb1c9bbad4492ea6b6195a9c732c5f WatchSource:0}: Error finding container d46771b77e73219b39ac090682c1a4e8f2bb1c9bbad4492ea6b6195a9c732c5f: Status 404 returned error can't find the container with id d46771b77e73219b39ac090682c1a4e8f2bb1c9bbad4492ea6b6195a9c732c5f
Sep 17 15:38:35.845594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.845566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nrj2z"
Sep 17 15:38:35.853355 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:35.853334    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce38b5e_309f_42e2_a80b_43188cbbbf93.slice/crio-5e8d1ea8ecc411996053ed7af37717d22123454c44ebedfe823d27294de064f8 WatchSource:0}: Error finding container 5e8d1ea8ecc411996053ed7af37717d22123454c44ebedfe823d27294de064f8: Status 404 returned error can't find the container with id 5e8d1ea8ecc411996053ed7af37717d22123454c44ebedfe823d27294de064f8
Sep 17 15:38:35.859830 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:35.859811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:38:35.868058 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:35.868037    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c82c64_126e_4dce_9f44_a1279a7de1e4.slice/crio-55783ec81b3da86d0d7a76ab8da5e9c8371416f413ad227edd38ab73c4ee7336 WatchSource:0}: Error finding container 55783ec81b3da86d0d7a76ab8da5e9c8371416f413ad227edd38ab73c4ee7336: Status 404 returned error can't find the container with id 55783ec81b3da86d0d7a76ab8da5e9c8371416f413ad227edd38ab73c4ee7336
Sep 17 15:38:36.998580 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:36.998555    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d8v86"
Sep 17 15:38:37.008069 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:37.008048    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29378f5b_d5cc_4654_ac7c_f0baf5c85832.slice/crio-7d6b724aa499d43310b7db29d36e7e6de21e017f9acc5ec0f786f50f64b59883 WatchSource:0}: Error finding container 7d6b724aa499d43310b7db29d36e7e6de21e017f9acc5ec0f786f50f64b59883: Status 404 returned error can't find the container with id 7d6b724aa499d43310b7db29d36e7e6de21e017f9acc5ec0f786f50f64b59883
Sep 17 15:38:37.012453 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.012435    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:37.026691 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.026676    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8t2t7"
Sep 17 15:38:37.032635 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.032620    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:38:37.034793 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:37.034774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0814fe57_0b03_485c_bd00_544675e5ec2c.slice/crio-1eba5772e1ee6d40f43de3eb72369dfd4690435f4fe18b173461620cb1fcaca1 WatchSource:0}: Error finding container 1eba5772e1ee6d40f43de3eb72369dfd4690435f4fe18b173461620cb1fcaca1: Status 404 returned error can't find the container with id 1eba5772e1ee6d40f43de3eb72369dfd4690435f4fe18b173461620cb1fcaca1
Sep 17 15:38:37.040858 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:38:37.040838    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce14c586_76d1_4bf1_8f04_f261b9afc9ce.slice/crio-6c664b39ec25f8276505bdddc65f6c0d65de9e70b1579b169919f132d3f1789c WatchSource:0}: Error finding container 6c664b39ec25f8276505bdddc65f6c0d65de9e70b1579b169919f132d3f1789c: Status 404 returned error can't find the container with id 6c664b39ec25f8276505bdddc65f6c0d65de9e70b1579b169919f132d3f1789c
Sep 17 15:38:37.284769 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.284689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:37.284769 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.284741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:37.285005 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:37.284900    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:37.285005 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:37.284967    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret podName:b4624302-fc21-4d0e-8c9a-fdfd494389b3 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:38.284953266 +0000 UTC m=+3.127316048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret") pod "global-pull-secret-syncer-j755r" (UID: "b4624302-fc21-4d0e-8c9a-fdfd494389b3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:37.285256 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:37.284901    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:37.285256 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:37.285216    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs podName:d86c0b23-dea0-497e-a3ca-fa1037439372 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:38.28519265 +0000 UTC m=+3.127555448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs") pod "network-metrics-daemon-xztjr" (UID: "d86c0b23-dea0-497e-a3ca-fa1037439372") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:37.384382 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.384347    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:37.385585 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.385559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:37.385777 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:37.385759    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:37.385845 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:37.385785    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:37.385845 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:37.385814    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s7jmc for pod openshift-network-diagnostics/network-check-target-jw9rz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:37.385917 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:37.385869    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc podName:da28f9b8-8cc1-45d7-a6b7-aebff23cc910 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:38.385852082 +0000 UTC m=+3.228214880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7jmc" (UniqueName: "kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc") pod "network-check-target-jw9rz" (UID: "da28f9b8-8cc1-45d7-a6b7-aebff23cc910") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:37.612380 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.612251    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:33:36 +0000 UTC" deadline="2028-06-18 11:55:46.919133678 +0000 UTC"
Sep 17 15:38:37.612380 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.612290    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15356h17m9.30684771s"
Sep 17 15:38:37.661496 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.661466    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:37.717571 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.717512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl" event={"ID":"9ff7e04e-bc8e-47db-aa47-09030f87b97a","Type":"ContainerStarted","Data":"854cda563396481dc986e56ff2c3c20c31b7d1c99c7c26d5ec9a054514cb2487"}
Sep 17 15:38:37.737154 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.737117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal" event={"ID":"37310a8518ae4b4c8f438099a305077e","Type":"ContainerStarted","Data":"cc1de874528080a4521b81b3161de59e35e18bf4d4e785d1f3863d2933fa6bf2"}
Sep 17 15:38:37.756772 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.756731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal" event={"ID":"0a022f343c02e056acab4e9c92407e8c","Type":"ContainerStarted","Data":"8ed5fa4dbd4391da69039813dee30527d011e157947cdfaa5b891fbcba9a7ad6"}
Sep 17 15:38:37.770356 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.768259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" event={"ID":"ce14c586-76d1-4bf1-8f04-f261b9afc9ce","Type":"ContainerStarted","Data":"6c664b39ec25f8276505bdddc65f6c0d65de9e70b1579b169919f132d3f1789c"}
Sep 17 15:38:37.788253 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.788181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nrj2z" event={"ID":"4ce38b5e-309f-42e2-a80b-43188cbbbf93","Type":"ContainerStarted","Data":"5e8d1ea8ecc411996053ed7af37717d22123454c44ebedfe823d27294de064f8"}
Sep 17 15:38:37.806893 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.806850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rwm2w" event={"ID":"62b44d6a-61ff-458c-917c-0f4a105a381d","Type":"ContainerStarted","Data":"55861f0c966f822007a0c83ba74053a528883ab28312355b1765fa6ce34a5d53"}
Sep 17 15:38:37.816440 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.816384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hv6gp" event={"ID":"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b","Type":"ContainerStarted","Data":"147e58cf6cee11c1abce8a611a7e4dd4af91ce172efd003fbf3f54a7bbdf0bbe"}
Sep 17 15:38:37.829546 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.829506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2t7" event={"ID":"0814fe57-0b03-485c-bd00-544675e5ec2c","Type":"ContainerStarted","Data":"1eba5772e1ee6d40f43de3eb72369dfd4690435f4fe18b173461620cb1fcaca1"}
Sep 17 15:38:37.833177 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.833144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d8v86" event={"ID":"29378f5b-d5cc-4654-ac7c-f0baf5c85832","Type":"ContainerStarted","Data":"7d6b724aa499d43310b7db29d36e7e6de21e017f9acc5ec0f786f50f64b59883"}
Sep 17 15:38:37.862655 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.862548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88mrl" event={"ID":"a7c82c64-126e-4dce-9f44-a1279a7de1e4","Type":"ContainerStarted","Data":"55783ec81b3da86d0d7a76ab8da5e9c8371416f413ad227edd38ab73c4ee7336"}
Sep 17 15:38:37.875282 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:37.875244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5fm5g" event={"ID":"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2","Type":"ContainerStarted","Data":"d46771b77e73219b39ac090682c1a4e8f2bb1c9bbad4492ea6b6195a9c732c5f"}
Sep 17 15:38:38.294580 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:38.294491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:38.294580 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:38.294549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:38.294831 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.294665    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:38.294831 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.294724    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs podName:d86c0b23-dea0-497e-a3ca-fa1037439372 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:40.294706368 +0000 UTC m=+5.137069167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs") pod "network-metrics-daemon-xztjr" (UID: "d86c0b23-dea0-497e-a3ca-fa1037439372") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:38.295170 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.295148    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:38.295234 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.295205    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret podName:b4624302-fc21-4d0e-8c9a-fdfd494389b3 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:40.295190754 +0000 UTC m=+5.137553550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret") pod "global-pull-secret-syncer-j755r" (UID: "b4624302-fc21-4d0e-8c9a-fdfd494389b3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:38.395127 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:38.395092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:38.395310 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.395252    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:38.395310 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.395273    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:38.395310 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.395285    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s7jmc for pod openshift-network-diagnostics/network-check-target-jw9rz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:38.395475 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.395351    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc podName:da28f9b8-8cc1-45d7-a6b7-aebff23cc910 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:40.395325302 +0000 UTC m=+5.237688184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7jmc" (UniqueName: "kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc") pod "network-check-target-jw9rz" (UID: "da28f9b8-8cc1-45d7-a6b7-aebff23cc910") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:38.612978 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:38.612778    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:33:36 +0000 UTC" deadline="2028-04-10 05:29:22.411774872 +0000 UTC"
Sep 17 15:38:38.612978 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:38.612857    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13693h50m43.798921994s"
Sep 17 15:38:38.669455 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:38.669419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:38.669644 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.669558    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:38.669991 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:38.669970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:38.670104 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.670085    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:38.670165 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:38.670158    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:38.670266 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:38.670225    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:40.315031 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:40.314697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:40.315515 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:40.315066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:40.315515 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.314857    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:40.315515 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.315192    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret podName:b4624302-fc21-4d0e-8c9a-fdfd494389b3 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:44.315167447 +0000 UTC m=+9.157530233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret") pod "global-pull-secret-syncer-j755r" (UID: "b4624302-fc21-4d0e-8c9a-fdfd494389b3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:40.315515 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.315250    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:40.315515 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.315302    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs podName:d86c0b23-dea0-497e-a3ca-fa1037439372 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:44.31528754 +0000 UTC m=+9.157650330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs") pod "network-metrics-daemon-xztjr" (UID: "d86c0b23-dea0-497e-a3ca-fa1037439372") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:40.416357 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:40.416326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:40.416537 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.416508    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:40.416537 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.416525    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:40.416537 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.416535    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s7jmc for pod openshift-network-diagnostics/network-check-target-jw9rz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:40.416659 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.416589    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc podName:da28f9b8-8cc1-45d7-a6b7-aebff23cc910 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:44.416576906 +0000 UTC m=+9.258939701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7jmc" (UniqueName: "kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc") pod "network-check-target-jw9rz" (UID: "da28f9b8-8cc1-45d7-a6b7-aebff23cc910") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:40.670064 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:40.669986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:40.670203 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.670119    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:40.670511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:40.670483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:40.670609 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.670589    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:40.670663 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:40.670640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:40.670733 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:40.670703    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:42.669882 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:42.669848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:42.670383 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:42.669987    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:42.670383 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:42.670381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:42.670509 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:42.670483    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:42.670568 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:42.670532    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:42.670625 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:42.670607    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:44.349611 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:44.349565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:44.350091 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:44.349628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:44.350091 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.349752    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:44.350091 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.349789    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:44.350091 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.349847    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret podName:b4624302-fc21-4d0e-8c9a-fdfd494389b3 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:52.349825522 +0000 UTC m=+17.192188312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret") pod "global-pull-secret-syncer-j755r" (UID: "b4624302-fc21-4d0e-8c9a-fdfd494389b3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:44.350091 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.349871    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs podName:d86c0b23-dea0-497e-a3ca-fa1037439372 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:52.349860104 +0000 UTC m=+17.192222888 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs") pod "network-metrics-daemon-xztjr" (UID: "d86c0b23-dea0-497e-a3ca-fa1037439372") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:44.450132 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:44.450039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:44.450301 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.450229    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:44.450301 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.450255    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:44.450301 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.450268    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s7jmc for pod openshift-network-diagnostics/network-check-target-jw9rz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:44.450444 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.450330    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc podName:da28f9b8-8cc1-45d7-a6b7-aebff23cc910 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:52.450310627 +0000 UTC m=+17.292673427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7jmc" (UniqueName: "kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc") pod "network-check-target-jw9rz" (UID: "da28f9b8-8cc1-45d7-a6b7-aebff23cc910") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:44.670014 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:44.669982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:44.670199 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.670095    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:44.670435 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:44.670411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:44.670544 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.670492    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:44.670544 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:44.670536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:44.670622 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:44.670579    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:46.669725 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:46.669259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:46.669725 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:46.669299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:46.669725 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:46.669372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:46.669725 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:46.669388    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:46.669725 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:46.669457    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:46.669725 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:46.669537    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:48.669045 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:48.669015    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:48.669045 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:48.669038    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:48.669470 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:48.669015    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:48.669470 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:48.669118    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:48.669470 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:48.669278    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:48.669470 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:48.669385    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:50.669546 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:50.669512    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:50.669933 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:50.669621    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:50.669933 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:50.669512    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:50.669933 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:50.669687    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:50.669933 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:50.669512    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:50.669933 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:50.669735    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:52.407905 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:52.407868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:52.408421 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:52.407918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:52.408421 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.408042    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:52.408421 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.408064    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:52.408421 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.408120    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs podName:d86c0b23-dea0-497e-a3ca-fa1037439372 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:08.408101518 +0000 UTC m=+33.250464301 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs") pod "network-metrics-daemon-xztjr" (UID: "d86c0b23-dea0-497e-a3ca-fa1037439372") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:52.408421 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.408137    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret podName:b4624302-fc21-4d0e-8c9a-fdfd494389b3 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:08.408129831 +0000 UTC m=+33.250492614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret") pod "global-pull-secret-syncer-j755r" (UID: "b4624302-fc21-4d0e-8c9a-fdfd494389b3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:52.508775 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:52.508742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:52.508977 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.508909    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:52.508977 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.508927    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:52.508977 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.508937    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s7jmc for pod openshift-network-diagnostics/network-check-target-jw9rz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:52.509100 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.508987    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc podName:da28f9b8-8cc1-45d7-a6b7-aebff23cc910 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:08.508973015 +0000 UTC m=+33.351335801 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7jmc" (UniqueName: "kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc") pod "network-check-target-jw9rz" (UID: "da28f9b8-8cc1-45d7-a6b7-aebff23cc910") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:52.669940 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:52.669858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:52.670082 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:52.669861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:52.670082 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.669992    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:52.670082 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:52.669861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:52.670234 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.670055    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:52.670234 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:52.670170    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:54.670055 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:54.670021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:54.670621 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:54.670063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:54.670621 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:54.670130    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:54.670621 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:54.670147    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:54.670621 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:54.670215    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:54.670621 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:54.670302    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:56.673004 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.670250    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:56.673004 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:56.670727    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:56.673004 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.670921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:56.673004 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:56.671014    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:56.673004 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.671111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:56.673004 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:56.671181    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:56.931988 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.931954    2580 generic.go:356] "Generic (PLEG): container finished" podID="0814fe57-0b03-485c-bd00-544675e5ec2c" containerID="289b2185d250e54150a122c8928557c4b2f708fdc00e7738c1204b14eeac3808" exitCode=0
Sep 17 15:38:56.932153 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.931991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2t7" event={"ID":"0814fe57-0b03-485c-bd00-544675e5ec2c","Type":"ContainerDied","Data":"289b2185d250e54150a122c8928557c4b2f708fdc00e7738c1204b14eeac3808"}
Sep 17 15:38:56.933840 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.933792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d8v86" event={"ID":"29378f5b-d5cc-4654-ac7c-f0baf5c85832","Type":"ContainerStarted","Data":"20579f750bbdac1b2dca46cc4b1540e034fb78434e647e9656d4db249e6c9a8a"}
Sep 17 15:38:56.935333 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.935303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88mrl" event={"ID":"a7c82c64-126e-4dce-9f44-a1279a7de1e4","Type":"ContainerStarted","Data":"1435d3f1d05ed1afa2293e8b70d2366cdc250758e135d555dc5695c96971a4f0"}
Sep 17 15:38:56.936901 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.936874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5fm5g" event={"ID":"8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2","Type":"ContainerStarted","Data":"84288ae50106deb6ddb319792aa00bebb3b054a61d5a0c7d2ce51b1539f0b96b"}
Sep 17 15:38:56.938475 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.938451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl" event={"ID":"9ff7e04e-bc8e-47db-aa47-09030f87b97a","Type":"ContainerStarted","Data":"d38d43a38f3d90687c922ec2145c0cdc046afb152b14d09ff2d683bf15818937"}
Sep 17 15:38:56.939790 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.939757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal" event={"ID":"37310a8518ae4b4c8f438099a305077e","Type":"ContainerStarted","Data":"43c54451c9889bd703f693d471f93a425b4a9b8d43da860c621e74093db5f33c"}
Sep 17 15:38:56.941266 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.941247    2580 generic.go:356] "Generic (PLEG): container finished" podID="0a022f343c02e056acab4e9c92407e8c" containerID="a030b9393683efed8514e6c086d583a21bd5409ffc4293cbc5ac22ccd98efcd8" exitCode=0
Sep 17 15:38:56.941373 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.941311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal" event={"ID":"0a022f343c02e056acab4e9c92407e8c","Type":"ContainerDied","Data":"a030b9393683efed8514e6c086d583a21bd5409ffc4293cbc5ac22ccd98efcd8"}
Sep 17 15:38:56.944609 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.944577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" event={"ID":"ce14c586-76d1-4bf1-8f04-f261b9afc9ce","Type":"ContainerStarted","Data":"8fd53ae8baa576e590f0b8bf8bbdb349a4aa65c565f51b9d76bae6829b9a0c0b"}
Sep 17 15:38:56.944694 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.944647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" event={"ID":"ce14c586-76d1-4bf1-8f04-f261b9afc9ce","Type":"ContainerStarted","Data":"4846b92c6fdadeffe64b5e91642e8dbbdc77a9744e0150ca7bb0f23ba575547c"}
Sep 17 15:38:56.944735 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.944698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" event={"ID":"ce14c586-76d1-4bf1-8f04-f261b9afc9ce","Type":"ContainerStarted","Data":"536e304c251ea78f8aea4985c57accbeaae4daacc160144eb5947faed20a8cc7"}
Sep 17 15:38:56.944781 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.944745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" event={"ID":"ce14c586-76d1-4bf1-8f04-f261b9afc9ce","Type":"ContainerStarted","Data":"3e8d7477705694232caee13f2fefeadf26f5a3876bef831516cb1ec7c52ef206"}
Sep 17 15:38:56.944856 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.944790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" event={"ID":"ce14c586-76d1-4bf1-8f04-f261b9afc9ce","Type":"ContainerStarted","Data":"f30355696553f1ca51956077b9fde3dcefa3b58317bea953127738555e29725d"}
Sep 17 15:38:56.944907 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.944869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" event={"ID":"ce14c586-76d1-4bf1-8f04-f261b9afc9ce","Type":"ContainerStarted","Data":"af18825a9f686881de1c08e8db58d8ecee07432689c1b5aa660e44bc35162981"}
Sep 17 15:38:56.946186 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.946166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rwm2w" event={"ID":"62b44d6a-61ff-458c-917c-0f4a105a381d","Type":"ContainerStarted","Data":"64a3db7b929d88ff476cb9e33de2a073d8a0c2a75caac8f716d2a58fb03f67b1"}
Sep 17 15:38:56.947822 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.947784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hv6gp" event={"ID":"6ecaf429-99cd-48a9-8eaf-96cdc04dd17b","Type":"ContainerStarted","Data":"18866ac314944b62056ca8996b75572f5eed866cb3bc9bcd01451c1c8385ecf2"}
Sep 17 15:38:56.973358 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.973311    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal" podStartSLOduration=20.973296674 podStartE2EDuration="20.973296674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:56.973001974 +0000 UTC m=+21.815364780" watchObservedRunningTime="2026-09-17 15:38:56.973296674 +0000 UTC m=+21.815659479"
Sep 17 15:38:56.986976 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:56.986936    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hv6gp" podStartSLOduration=2.714518259 podStartE2EDuration="21.986926528s" totalImagesPullingTime="19.272408269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:56.986410187 +0000 UTC m=+21.828772993" watchObservedRunningTime="2026-09-17 15:38:56.986926528 +0000 UTC m=+21.829289333"
Sep 17 15:38:57.000277 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.000239    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rwm2w" podStartSLOduration=2.765311059 podStartE2EDuration="22.000225003s" totalImagesPullingTime="19.234913944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:56.999891468 +0000 UTC m=+21.842254272" watchObservedRunningTime="2026-09-17 15:38:57.000225003 +0000 UTC m=+21.842587836"
Sep 17 15:38:57.021201 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.021167    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5fm5g" podStartSLOduration=2.7360162519999998 podStartE2EDuration="22.02115388s" totalImagesPullingTime="19.285137628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:57.020772718 +0000 UTC m=+21.863135523" watchObservedRunningTime="2026-09-17 15:38:57.02115388 +0000 UTC m=+21.863516685"
Sep 17 15:38:57.055606 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.055506    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-88mrl" podStartSLOduration=10.520097405 podStartE2EDuration="22.055485488s" totalImagesPullingTime="11.535388083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:57.054512894 +0000 UTC m=+21.896875702" watchObservedRunningTime="2026-09-17 15:38:57.055485488 +0000 UTC m=+21.897848297"
Sep 17 15:38:57.074614 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.074576    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d8v86" podStartSLOduration=2.8918833299999998 podStartE2EDuration="22.07456256s" totalImagesPullingTime="19.18267923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:57.074456319 +0000 UTC m=+21.916819126" watchObservedRunningTime="2026-09-17 15:38:57.07456256 +0000 UTC m=+21.916925365"
Sep 17 15:38:57.752482 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.752454    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:38:57.952970 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.952926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl" event={"ID":"9ff7e04e-bc8e-47db-aa47-09030f87b97a","Type":"ContainerStarted","Data":"f78297f5bb17e3185f2b3a51ba225ba8770ef733dde2f953b5ccf171f1345762"}
Sep 17 15:38:57.954808 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.954776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal" event={"ID":"0a022f343c02e056acab4e9c92407e8c","Type":"ContainerStarted","Data":"4678aadc08295bb2c516fb7536ae6fe9f776cf483ee131683d2d050fa767aa7a"}
Sep 17 15:38:57.956471 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.956444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nrj2z" event={"ID":"4ce38b5e-309f-42e2-a80b-43188cbbbf93","Type":"ContainerStarted","Data":"986b08a59ba0cfd2fde8eec7d2d0c25075da49f5b1acbd6d0b07769503c5e388"}
Sep 17 15:38:57.970074 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.970036    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal" podStartSLOduration=21.970023396 podStartE2EDuration="21.970023396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:57.9692661 +0000 UTC m=+22.811628907" watchObservedRunningTime="2026-09-17 15:38:57.970023396 +0000 UTC m=+22.812386202"
Sep 17 15:38:57.987363 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:57.987311    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nrj2z" podStartSLOduration=3.768790598 podStartE2EDuration="22.987292643s" totalImagesPullingTime="19.218502045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:57.986540452 +0000 UTC m=+22.828903281" watchObservedRunningTime="2026-09-17 15:38:57.987292643 +0000 UTC m=+22.829655450"
Sep 17 15:38:58.656059 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:58.655945    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:38:57.752476106Z","UUID":"c8ae4514-575a-480b-89d2-85ab6450ca60","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:38:58.657767 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:58.657714    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:38:58.657767 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:58.657750    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:38:58.669297 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:58.669272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:38:58.669404 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:58.669374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:38:58.669404 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:58.669387    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:38:58.669523 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:58.669447    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:38:58.669523 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:58.669489    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:38:58.669644 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:38:58.669617    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:38:58.961188 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:58.961159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl" event={"ID":"9ff7e04e-bc8e-47db-aa47-09030f87b97a","Type":"ContainerStarted","Data":"fbb26fb62df4b4a0f3fb6a7946a9d91af2033ee6ac7f84cf880636f60a185e9c"}
Sep 17 15:38:58.979273 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:58.979233    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5svpl" podStartSLOduration=2.934717573 podStartE2EDuration="23.979217259s" totalImagesPullingTime="21.044499686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:58.978691603 +0000 UTC m=+23.821054408" watchObservedRunningTime="2026-09-17 15:38:58.979217259 +0000 UTC m=+23.821580063"
Sep 17 15:38:59.969677 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:38:59.969636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" event={"ID":"ce14c586-76d1-4bf1-8f04-f261b9afc9ce","Type":"ContainerStarted","Data":"0deb745a7a05f3ba81f9f8f76e0b0c0f86a13863643623f1cf2ab6dce4d17aff"}
Sep 17 15:39:00.669636 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:00.669453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:00.669832 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:00.669453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:39:00.669832 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:00.669710    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:39:00.669832 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:00.669453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:00.669832 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:00.669812    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:39:00.670045 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:00.669857    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:39:01.104486 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.104448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:39:01.105366 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.105347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:39:01.940781 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.940758    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:01.976044 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.975988    2580 generic.go:356] "Generic (PLEG): container finished" podID="0814fe57-0b03-485c-bd00-544675e5ec2c" containerID="cb9d85385ce98ec34dbc33f0af5f4cb00b9db6cc31c33026106572a2abfe9899" exitCode=0
Sep 17 15:39:01.976146 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.976068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2t7" event={"ID":"0814fe57-0b03-485c-bd00-544675e5ec2c","Type":"ContainerDied","Data":"cb9d85385ce98ec34dbc33f0af5f4cb00b9db6cc31c33026106572a2abfe9899"}
Sep 17 15:39:01.980057 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.980023    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" event={"ID":"ce14c586-76d1-4bf1-8f04-f261b9afc9ce","Type":"ContainerStarted","Data":"20bd4f343b9a358c3168297f19878081bc7662b69a338cd6f9ab88b66683084a"}
Sep 17 15:39:01.980433 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.980413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:39:01.980539 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.980452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:39:01.980721 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.980705    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-88mrl"
Sep 17 15:39:01.995304 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:01.995283    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:39:02.027831 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:02.027771    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl" podStartSLOduration=7.771101188 podStartE2EDuration="27.027759221s" totalImagesPullingTime="19.256658033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:02.027110877 +0000 UTC m=+26.869473704" watchObservedRunningTime="2026-09-17 15:39:02.027759221 +0000 UTC m=+26.870122026"
Sep 17 15:39:02.032380 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:02.032360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:02.132999 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:02.132968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:02.669448 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:02.669268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:02.669766 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:02.669341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:39:02.669766 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:02.669540    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:39:02.669766 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:02.669351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:02.669766 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:02.669591    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:39:02.669766 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:02.669649    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:39:02.981964 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:02.981938    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:39:02.982451 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:02.982422    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:39:02.997676 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:02.997658    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:39:03.174430 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:03.174366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:39:03.406585 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:03.406553    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j755r"]
Sep 17 15:39:03.406753 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:03.406640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:03.406828 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:03.406747    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:39:03.409954 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:03.409930    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xztjr"]
Sep 17 15:39:03.410090 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:03.410031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:39:03.410171 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:03.410146    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:39:03.410611 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:03.410589    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jw9rz"]
Sep 17 15:39:03.410697 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:03.410675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:03.410790 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:03.410768    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:39:03.985447 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:03.985406    2580 generic.go:356] "Generic (PLEG): container finished" podID="0814fe57-0b03-485c-bd00-544675e5ec2c" containerID="1441c291fd19a7d922bbb9ff733eb5aab82417371df774d8864677d6e48fac92" exitCode=0
Sep 17 15:39:03.985634 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:03.985483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2t7" event={"ID":"0814fe57-0b03-485c-bd00-544675e5ec2c","Type":"ContainerDied","Data":"1441c291fd19a7d922bbb9ff733eb5aab82417371df774d8864677d6e48fac92"}
Sep 17 15:39:04.669997 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:04.669958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:04.670549 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:04.670072    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:39:04.670549 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:04.670132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:39:04.670549 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:04.670272    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:39:04.990339 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:04.990310    2580 generic.go:356] "Generic (PLEG): container finished" podID="0814fe57-0b03-485c-bd00-544675e5ec2c" containerID="4e8535eb8e00718c1201dfd91124890ce29ac046a8ade554f49a25c672dbdd76" exitCode=0
Sep 17 15:39:04.990464 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:04.990401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2t7" event={"ID":"0814fe57-0b03-485c-bd00-544675e5ec2c","Type":"ContainerDied","Data":"4e8535eb8e00718c1201dfd91124890ce29ac046a8ade554f49a25c672dbdd76"}
Sep 17 15:39:05.669908 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:05.669741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:05.670038 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:05.669971    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:39:06.669851 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:06.669819    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:39:06.670026 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:06.669820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:06.670026 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:06.669968    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:39:06.670026 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:06.669996    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:39:06.851088 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:06.851032    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:06.962030 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:06.961946    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:07.022640 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:07.022581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:07.669514 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:07.669440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:07.669662 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:07.669579    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j755r" podUID="b4624302-fc21-4d0e-8c9a-fdfd494389b3"
Sep 17 15:39:08.423106 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:08.423064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:08.423540 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:08.423122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:39:08.423540 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.423220    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:08.423540 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.423268    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:08.423540 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.423295    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret podName:b4624302-fc21-4d0e-8c9a-fdfd494389b3 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:40.423277787 +0000 UTC m=+65.265640588 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret") pod "global-pull-secret-syncer-j755r" (UID: "b4624302-fc21-4d0e-8c9a-fdfd494389b3") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:08.423540 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.423327    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs podName:d86c0b23-dea0-497e-a3ca-fa1037439372 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:40.423304373 +0000 UTC m=+65.265667160 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs") pod "network-metrics-daemon-xztjr" (UID: "d86c0b23-dea0-497e-a3ca-fa1037439372") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:08.524164 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:08.524120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:08.524340 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.524294    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:08.524340 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.524315    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:08.524340 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.524325    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s7jmc for pod openshift-network-diagnostics/network-check-target-jw9rz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:08.524486 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.524376    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc podName:da28f9b8-8cc1-45d7-a6b7-aebff23cc910 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:40.524363324 +0000 UTC m=+65.366726107 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7jmc" (UniqueName: "kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc") pod "network-check-target-jw9rz" (UID: "da28f9b8-8cc1-45d7-a6b7-aebff23cc910") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:08.669813 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:08.669781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:39:08.669983 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:08.669839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:08.669983 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.669918    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xztjr" podUID="d86c0b23-dea0-497e-a3ca-fa1037439372"
Sep 17 15:39:08.670076 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:08.670034    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jw9rz" podUID="da28f9b8-8cc1-45d7-a6b7-aebff23cc910"
Sep 17 15:39:08.961685 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:08.961655    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeReady"
Sep 17 15:39:08.961859 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:08.961815    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:39:09.010140 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.010104    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bdtrg"]
Sep 17 15:39:09.043649 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.043622    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zzkfw"]
Sep 17 15:39:09.043836 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.043763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.045843 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.045818    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:39:09.045843 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.045830    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:39:09.046016 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.045821    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghpsh\""
Sep 17 15:39:09.059721 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.059667    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bdtrg"]
Sep 17 15:39:09.059829 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.059746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zzkfw"
Sep 17 15:39:09.059829 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.059761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zzkfw"]
Sep 17 15:39:09.062308 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.062282    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r2vb9\""
Sep 17 15:39:09.062308 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.062288    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:39:09.062534 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.062519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:39:09.063026 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.063008    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:39:09.128871 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.128842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.128871 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.128869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-tmp-dir\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.129047 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.128893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-config-volume\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.129047 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.128909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08c92ab5-a08c-4d43-9cb6-d20f69be2a2c-cert\") pod \"ingress-canary-zzkfw\" (UID: \"08c92ab5-a08c-4d43-9cb6-d20f69be2a2c\") " pod="openshift-ingress-canary/ingress-canary-zzkfw"
Sep 17 15:39:09.129047 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.128966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdnb\" (UniqueName: \"kubernetes.io/projected/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-kube-api-access-hxdnb\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.129047 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.129007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxl8\" (UniqueName: \"kubernetes.io/projected/08c92ab5-a08c-4d43-9cb6-d20f69be2a2c-kube-api-access-qnxl8\") pod \"ingress-canary-zzkfw\" (UID: \"08c92ab5-a08c-4d43-9cb6-d20f69be2a2c\") " pod="openshift-ingress-canary/ingress-canary-zzkfw"
Sep 17 15:39:09.229699 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.229667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-config-volume\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.229873 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.229709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08c92ab5-a08c-4d43-9cb6-d20f69be2a2c-cert\") pod \"ingress-canary-zzkfw\" (UID: \"08c92ab5-a08c-4d43-9cb6-d20f69be2a2c\") " pod="openshift-ingress-canary/ingress-canary-zzkfw"
Sep 17 15:39:09.229873 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.229724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdnb\" (UniqueName: \"kubernetes.io/projected/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-kube-api-access-hxdnb\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.229873 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.229758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxl8\" (UniqueName: \"kubernetes.io/projected/08c92ab5-a08c-4d43-9cb6-d20f69be2a2c-kube-api-access-qnxl8\") pod \"ingress-canary-zzkfw\" (UID: \"08c92ab5-a08c-4d43-9cb6-d20f69be2a2c\") " pod="openshift-ingress-canary/ingress-canary-zzkfw"
Sep 17 15:39:09.229873 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.229869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.230076 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.229892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-tmp-dir\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.230288 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.230146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-tmp-dir\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.230288 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:09.230234    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:09.230288 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:09.230282    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls podName:38dc1beb-42f8-4732-bc67-a5a9f39a97e6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:09.730266186 +0000 UTC m=+34.572628984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls") pod "dns-default-bdtrg" (UID: "38dc1beb-42f8-4732-bc67-a5a9f39a97e6") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:09.230288 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.230280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-config-volume\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.234494 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.234469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08c92ab5-a08c-4d43-9cb6-d20f69be2a2c-cert\") pod \"ingress-canary-zzkfw\" (UID: \"08c92ab5-a08c-4d43-9cb6-d20f69be2a2c\") " pod="openshift-ingress-canary/ingress-canary-zzkfw"
Sep 17 15:39:09.239341 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.239304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdnb\" (UniqueName: \"kubernetes.io/projected/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-kube-api-access-hxdnb\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.239436 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.239366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxl8\" (UniqueName: \"kubernetes.io/projected/08c92ab5-a08c-4d43-9cb6-d20f69be2a2c-kube-api-access-qnxl8\") pod \"ingress-canary-zzkfw\" (UID: \"08c92ab5-a08c-4d43-9cb6-d20f69be2a2c\") " pod="openshift-ingress-canary/ingress-canary-zzkfw"
Sep 17 15:39:09.369157 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.369128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zzkfw"
Sep 17 15:39:09.669504 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.669422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:09.682714 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.682689    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zzkfw"]
Sep 17 15:39:09.686217 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:39:09.686189    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c92ab5_a08c_4d43_9cb6_d20f69be2a2c.slice/crio-29dcd8599afef4cb764b6133a28e71ac7cbef658b30a8d072221feeb33ae91a6 WatchSource:0}: Error finding container 29dcd8599afef4cb764b6133a28e71ac7cbef658b30a8d072221feeb33ae91a6: Status 404 returned error can't find the container with id 29dcd8599afef4cb764b6133a28e71ac7cbef658b30a8d072221feeb33ae91a6
Sep 17 15:39:09.688509 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.688480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:39:09.734753 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:09.734732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:09.734903 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:09.734887    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:09.734956 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:09.734941    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls podName:38dc1beb-42f8-4732-bc67-a5a9f39a97e6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:10.734926596 +0000 UTC m=+35.577289378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls") pod "dns-default-bdtrg" (UID: "38dc1beb-42f8-4732-bc67-a5a9f39a97e6") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:10.004438 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.004242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zzkfw" event={"ID":"08c92ab5-a08c-4d43-9cb6-d20f69be2a2c","Type":"ContainerStarted","Data":"29dcd8599afef4cb764b6133a28e71ac7cbef658b30a8d072221feeb33ae91a6"}
Sep 17 15:39:10.007003 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.006974    2580 generic.go:356] "Generic (PLEG): container finished" podID="0814fe57-0b03-485c-bd00-544675e5ec2c" containerID="58f8d9b88e02db41b8f6da8001f2cace3bce1ada42000f7aad4402adc5864b36" exitCode=0
Sep 17 15:39:10.007091 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.007007    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2t7" event={"ID":"0814fe57-0b03-485c-bd00-544675e5ec2c","Type":"ContainerDied","Data":"58f8d9b88e02db41b8f6da8001f2cace3bce1ada42000f7aad4402adc5864b36"}
Sep 17 15:39:10.669329 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.669293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:10.669501 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.669303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:39:10.673033 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.673009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:39:10.673338 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.673047    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:39:10.673764 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.673742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k8wzk\""
Sep 17 15:39:10.673991 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.673976    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h48gq\""
Sep 17 15:39:10.674163 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.674143    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:39:10.744750 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:10.744555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:10.744750 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:10.744693    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:10.744750 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:10.744751    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls podName:38dc1beb-42f8-4732-bc67-a5a9f39a97e6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:12.744737619 +0000 UTC m=+37.587100402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls") pod "dns-default-bdtrg" (UID: "38dc1beb-42f8-4732-bc67-a5a9f39a97e6") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:11.013448 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:11.013364    2580 generic.go:356] "Generic (PLEG): container finished" podID="0814fe57-0b03-485c-bd00-544675e5ec2c" containerID="a6d4b94926dcde3a81238f59606bc38348f198e31d4a4dd8e821f79e6e7dccb5" exitCode=0
Sep 17 15:39:11.013448 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:11.013441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2t7" event={"ID":"0814fe57-0b03-485c-bd00-544675e5ec2c","Type":"ContainerDied","Data":"a6d4b94926dcde3a81238f59606bc38348f198e31d4a4dd8e821f79e6e7dccb5"}
Sep 17 15:39:12.020136 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:12.020097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8t2t7" event={"ID":"0814fe57-0b03-485c-bd00-544675e5ec2c","Type":"ContainerStarted","Data":"ff28cee9d8350d2fb4967fd56bb41dd818b106eb0c8eaae244471b3dffb36633"}
Sep 17 15:39:12.041811 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:12.041739    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8t2t7" podStartSLOduration=6.620290039 podStartE2EDuration="37.041721747s" totalImagesPullingTime="30.421431708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:12.04112433 +0000 UTC m=+36.883487136" watchObservedRunningTime="2026-09-17 15:39:12.041721747 +0000 UTC m=+36.884084551"
Sep 17 15:39:12.166104 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:12.166067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:12.262733 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:12.262705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:12.761177 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:12.761108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:12.761305 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:12.761213    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:12.761305 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:12.761270    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls podName:38dc1beb-42f8-4732-bc67-a5a9f39a97e6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:16.761257079 +0000 UTC m=+41.603619863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls") pod "dns-default-bdtrg" (UID: "38dc1beb-42f8-4732-bc67-a5a9f39a97e6") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:13.023517 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:13.023431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zzkfw" event={"ID":"08c92ab5-a08c-4d43-9cb6-d20f69be2a2c","Type":"ContainerStarted","Data":"794ef761a46fdc896bcd43a202ebd7d537a0d3536b792197d6e37df53b56ecdf"}
Sep 17 15:39:13.048477 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:13.048432    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zzkfw" podStartSLOduration=2.360626055 podStartE2EDuration="5.048412247s" totalImagesPullingTime="2.687786192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:13.047848038 +0000 UTC m=+37.890210853" watchObservedRunningTime="2026-09-17 15:39:13.048412247 +0000 UTC m=+37.890775088"
Sep 17 15:39:14.219430 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:14.219386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:15.065492 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:15.065455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:15.132388 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:15.132360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:15.254202 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:15.254166    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:15.324502 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:15.324423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:15.397429 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:15.397398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:15.497980 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:15.497919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:15.579397 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:15.579296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:16.059811 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:16.059780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:16.114269 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:16.114239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:16.194976 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:16.194952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:16.251528 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:16.251498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:16.591559 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:16.591529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:16.652470 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:16.652435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:16.791332 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:16.791275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:16.791503 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:16.791439    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:16.791542 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:16.791507    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls podName:38dc1beb-42f8-4732-bc67-a5a9f39a97e6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:24.7914893 +0000 UTC m=+49.633852083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls") pod "dns-default-bdtrg" (UID: "38dc1beb-42f8-4732-bc67-a5a9f39a97e6") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:24.843528 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:24.843488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:24.843958 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:24.843644    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:24.843958 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:24.843723    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls podName:38dc1beb-42f8-4732-bc67-a5a9f39a97e6 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:40.84370262 +0000 UTC m=+65.686065417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls") pod "dns-default-bdtrg" (UID: "38dc1beb-42f8-4732-bc67-a5a9f39a97e6") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:28.840756 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:28.840717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:30.124670 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:30.124641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:35.001446 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:35.001411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zpzbl"
Sep 17 15:39:35.149859 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:35.149828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:35.204636 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:35.204608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:36.147788 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:36.147750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:36.247541 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:36.247515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:36.347560 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:36.347530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:38.165213 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:38.165177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:38.263968 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:38.263935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:40.110839 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.110794    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:40.456286 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.456199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:40.456286 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.456243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:39:40.458765 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.458742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:39:40.458765 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.458757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:39:40.466481 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:40.466453    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:39:40.466538 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:40.466533    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs podName:d86c0b23-dea0-497e-a3ca-fa1037439372 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:44.466514961 +0000 UTC m=+129.308877757 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs") pod "network-metrics-daemon-xztjr" (UID: "d86c0b23-dea0-497e-a3ca-fa1037439372") : secret "metrics-daemon-secret" not found
Sep 17 15:39:40.469911 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.469887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4624302-fc21-4d0e-8c9a-fdfd494389b3-original-pull-secret\") pod \"global-pull-secret-syncer-j755r\" (UID: \"b4624302-fc21-4d0e-8c9a-fdfd494389b3\") " pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:40.556946 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.556905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:40.559366 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.559347    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:39:40.569954 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.569933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:39:40.578625 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.578598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j755r"
Sep 17 15:39:40.581159 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.581138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jmc\" (UniqueName: \"kubernetes.io/projected/da28f9b8-8cc1-45d7-a6b7-aebff23cc910-kube-api-access-s7jmc\") pod \"network-check-target-jw9rz\" (UID: \"da28f9b8-8cc1-45d7-a6b7-aebff23cc910\") " pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:40.683660 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.683632    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k8wzk\""
Sep 17 15:39:40.692329 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.692302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:40.858748 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.858714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:39:40.858907 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:40.858844    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:40.858958 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:39:40.858909    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls podName:38dc1beb-42f8-4732-bc67-a5a9f39a97e6 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:12.858893301 +0000 UTC m=+97.701256084 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls") pod "dns-default-bdtrg" (UID: "38dc1beb-42f8-4732-bc67-a5a9f39a97e6") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:40.883562 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.883534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j755r"]
Sep 17 15:39:40.886381 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:39:40.886357    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4624302_fc21_4d0e_8c9a_fdfd494389b3.slice/crio-35c9223c605489f528ed85c4bc5c94a770fbd587eea8a617f3159af9024a534d WatchSource:0}: Error finding container 35c9223c605489f528ed85c4bc5c94a770fbd587eea8a617f3159af9024a534d: Status 404 returned error can't find the container with id 35c9223c605489f528ed85c4bc5c94a770fbd587eea8a617f3159af9024a534d
Sep 17 15:39:40.974106 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:40.974016    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jw9rz"]
Sep 17 15:39:41.085258 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:41.085222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jw9rz" event={"ID":"da28f9b8-8cc1-45d7-a6b7-aebff23cc910","Type":"ContainerStarted","Data":"635d9d094a259443c6f01a8f22a5ec156885ec2e10a12ad027b3440c36cf0bb7"}
Sep 17 15:39:41.086263 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:41.086240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j755r" event={"ID":"b4624302-fc21-4d0e-8c9a-fdfd494389b3","Type":"ContainerStarted","Data":"35c9223c605489f528ed85c4bc5c94a770fbd587eea8a617f3159af9024a534d"}
Sep 17 15:39:42.075239 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:42.075210    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:42.137713 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:42.137681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:46.099897 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:46.099858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jw9rz" event={"ID":"da28f9b8-8cc1-45d7-a6b7-aebff23cc910","Type":"ContainerStarted","Data":"f09e0e6457cbdf9975af1d71ea0f0ebbf677143f14c5df090b2b4cb2f7c2cc3c"}
Sep 17 15:39:46.100328 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:46.099954    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:39:46.101302 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:46.101279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j755r" event={"ID":"b4624302-fc21-4d0e-8c9a-fdfd494389b3","Type":"ContainerStarted","Data":"97e1ec3efd8d15ed2690034607c52130c9fb49573fbc6e6a9d4d8efcb05fed65"}
Sep 17 15:39:46.117699 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:46.117655    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jw9rz" podStartSLOduration=66.673746657 podStartE2EDuration="1m11.117639658s" totalImagesPullingTime="4.443893001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:46.117229231 +0000 UTC m=+70.959592036" watchObservedRunningTime="2026-09-17 15:39:46.117639658 +0000 UTC m=+70.960002462"
Sep 17 15:39:48.811544 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:48.811512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:48.873420 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:48.873401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:51.238370 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:51.238325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:51.301134 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:51.301109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:39:53.835375 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:39:53.835342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:07.923208 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:07.923142    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j755r" podStartSLOduration=88.383742613 podStartE2EDuration="1m32.923099766s" totalImagesPullingTime="4.539357153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:46.130432314 +0000 UTC m=+70.972795119" watchObservedRunningTime="2026-09-17 15:40:07.923099766 +0000 UTC m=+92.765462575"
Sep 17 15:40:07.924007 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:07.923990    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"]
Sep 17 15:40:07.926699 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:07.926685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:07.928982 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:07.928957    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 15:40:07.929132 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:07.929026    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 15:40:07.929326 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:07.929305    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 15:40:07.929419 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:07.929405    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-b98rb\""
Sep 17 15:40:07.929594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:07.929580    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:40:07.937126 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:07.937104    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"]
Sep 17 15:40:08.028418 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.028391    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-9jg52"]
Sep 17 15:40:08.030113 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.030095    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.032645 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.032628    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 15:40:08.033070 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.033047    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:40:08.033170 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.033118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 15:40:08.033253 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.033199    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:40:08.033361 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.033346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-flxch\""
Sep 17 15:40:08.034736 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.034716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96dm\" (UniqueName: \"kubernetes.io/projected/a6762993-a05b-4825-8b15-c77c144e43e3-kube-api-access-b96dm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rl5dj\" (UID: \"a6762993-a05b-4825-8b15-c77c144e43e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.034858 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.034845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6762993-a05b-4825-8b15-c77c144e43e3-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rl5dj\" (UID: \"a6762993-a05b-4825-8b15-c77c144e43e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.034933 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.034878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6762993-a05b-4825-8b15-c77c144e43e3-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rl5dj\" (UID: \"a6762993-a05b-4825-8b15-c77c144e43e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.045699 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.045678    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 15:40:08.046575 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.046557    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9jg52"]
Sep 17 15:40:08.125765 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.125734    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4"]
Sep 17 15:40:08.127705 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.127691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4"
Sep 17 15:40:08.129465 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.129441    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-k7mhd"]
Sep 17 15:40:08.130725 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.130708    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 15:40:08.130970 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.130946    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:40:08.130970 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.130968    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-grmxl\""
Sep 17 15:40:08.131549 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.131532    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"]
Sep 17 15:40:08.131672 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.131656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.133188 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.133173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:08.133544 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.133529    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 15:40:08.133629 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.133530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-t4f8q\""
Sep 17 15:40:08.133629 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.133552    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 15:40:08.133862 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.133836    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:40:08.133862 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.133857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 15:40:08.134844 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.134823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 15:40:08.135158 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.135140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c959272-5175-4063-bf12-e0b9b8e38650-serving-cert\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.135254 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.135173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6762993-a05b-4825-8b15-c77c144e43e3-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rl5dj\" (UID: \"a6762993-a05b-4825-8b15-c77c144e43e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.135254 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.135192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6762993-a05b-4825-8b15-c77c144e43e3-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rl5dj\" (UID: \"a6762993-a05b-4825-8b15-c77c144e43e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.135254 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.135222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0c959272-5175-4063-bf12-e0b9b8e38650-snapshots\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.135863 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.135510    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 15:40:08.135961 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.135914    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vzkpq\""
Sep 17 15:40:08.136114 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.136094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c959272-5175-4063-bf12-e0b9b8e38650-service-ca-bundle\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.136178 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.136122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795b2\" (UniqueName: \"kubernetes.io/projected/0c959272-5175-4063-bf12-e0b9b8e38650-kube-api-access-795b2\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.136178 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.136143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c959272-5175-4063-bf12-e0b9b8e38650-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.136178 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.136147    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:40:08.136178 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.136163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96dm\" (UniqueName: \"kubernetes.io/projected/a6762993-a05b-4825-8b15-c77c144e43e3-kube-api-access-b96dm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rl5dj\" (UID: \"a6762993-a05b-4825-8b15-c77c144e43e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.136178 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.136178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c959272-5175-4063-bf12-e0b9b8e38650-tmp\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.136984 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.136845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a6762993-a05b-4825-8b15-c77c144e43e3-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rl5dj\" (UID: \"a6762993-a05b-4825-8b15-c77c144e43e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.137567 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.137549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4"]
Sep 17 15:40:08.138098 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.138080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a6762993-a05b-4825-8b15-c77c144e43e3-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rl5dj\" (UID: \"a6762993-a05b-4825-8b15-c77c144e43e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.139865 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.139848    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 15:40:08.142496 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.142473    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"]
Sep 17 15:40:08.143263 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.143245    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-k7mhd"]
Sep 17 15:40:08.146762 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.146743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96dm\" (UniqueName: \"kubernetes.io/projected/a6762993-a05b-4825-8b15-c77c144e43e3-kube-api-access-b96dm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rl5dj\" (UID: \"a6762993-a05b-4825-8b15-c77c144e43e3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.198855 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.198781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:08.234985 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.234966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c959272-5175-4063-bf12-e0b9b8e38650-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c959272-5175-4063-bf12-e0b9b8e38650-tmp\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf76r\" (UniqueName: \"kubernetes.io/projected/46a415f3-4b6c-42ef-acce-0b4d7d470631-kube-api-access-sf76r\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2037b600-0a62-4629-93a6-16b83614e117-serving-cert\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqr9z\" (UniqueName: \"kubernetes.io/projected/2b7ac916-0662-4e55-860d-da12c759654f-kube-api-access-bqr9z\") pod \"volume-data-source-validator-86695b7cc7-7xgj4\" (UID: \"2b7ac916-0662-4e55-860d-da12c759654f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c959272-5175-4063-bf12-e0b9b8e38650-serving-cert\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0c959272-5175-4063-bf12-e0b9b8e38650-snapshots\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2037b600-0a62-4629-93a6-16b83614e117-trusted-ca\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.236994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c959272-5175-4063-bf12-e0b9b8e38650-service-ca-bundle\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.237021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795b2\" (UniqueName: \"kubernetes.io/projected/0c959272-5175-4063-bf12-e0b9b8e38650-kube-api-access-795b2\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.237046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.237047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2037b600-0a62-4629-93a6-16b83614e117-config\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.237618 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.237066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c959272-5175-4063-bf12-e0b9b8e38650-tmp\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.237618 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.237072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvj7\" (UniqueName: \"kubernetes.io/projected/2037b600-0a62-4629-93a6-16b83614e117-kube-api-access-vlvj7\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.237618 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.237517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c959272-5175-4063-bf12-e0b9b8e38650-service-ca-bundle\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.237618 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.237522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c959272-5175-4063-bf12-e0b9b8e38650-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.237762 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.237742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0c959272-5175-4063-bf12-e0b9b8e38650-snapshots\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.239369 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.239341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c959272-5175-4063-bf12-e0b9b8e38650-serving-cert\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.245102 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.245083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-795b2\" (UniqueName: \"kubernetes.io/projected/0c959272-5175-4063-bf12-e0b9b8e38650-kube-api-access-795b2\") pod \"insights-operator-5844b856db-9jg52\" (UID: \"0c959272-5175-4063-bf12-e0b9b8e38650\") " pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.254576 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.254553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:08.338440 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.338402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2037b600-0a62-4629-93a6-16b83614e117-trusted-ca\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.338577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.338462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2037b600-0a62-4629-93a6-16b83614e117-config\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.338577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.338484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvj7\" (UniqueName: \"kubernetes.io/projected/2037b600-0a62-4629-93a6-16b83614e117-kube-api-access-vlvj7\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.338577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.338516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf76r\" (UniqueName: \"kubernetes.io/projected/46a415f3-4b6c-42ef-acce-0b4d7d470631-kube-api-access-sf76r\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:08.338577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.338539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2037b600-0a62-4629-93a6-16b83614e117-serving-cert\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.338577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.338561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqr9z\" (UniqueName: \"kubernetes.io/projected/2b7ac916-0662-4e55-860d-da12c759654f-kube-api-access-bqr9z\") pod \"volume-data-source-validator-86695b7cc7-7xgj4\" (UID: \"2b7ac916-0662-4e55-860d-da12c759654f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4"
Sep 17 15:40:08.338838 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.338622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:08.338838 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:08.338717    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:08.338838 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.338728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9jg52"
Sep 17 15:40:08.338838 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:08.338789    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls podName:46a415f3-4b6c-42ef-acce-0b4d7d470631 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:08.838769056 +0000 UTC m=+93.681131855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6dthc" (UID: "46a415f3-4b6c-42ef-acce-0b4d7d470631") : secret "samples-operator-tls" not found
Sep 17 15:40:08.339258 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.339240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2037b600-0a62-4629-93a6-16b83614e117-config\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.339334 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.339316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2037b600-0a62-4629-93a6-16b83614e117-trusted-ca\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.341008 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.340989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2037b600-0a62-4629-93a6-16b83614e117-serving-cert\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.347170 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.347148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqr9z\" (UniqueName: \"kubernetes.io/projected/2b7ac916-0662-4e55-860d-da12c759654f-kube-api-access-bqr9z\") pod \"volume-data-source-validator-86695b7cc7-7xgj4\" (UID: \"2b7ac916-0662-4e55-860d-da12c759654f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4"
Sep 17 15:40:08.347271 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.347207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvj7\" (UniqueName: \"kubernetes.io/projected/2037b600-0a62-4629-93a6-16b83614e117-kube-api-access-vlvj7\") pod \"console-operator-696884fb9c-k7mhd\" (UID: \"2037b600-0a62-4629-93a6-16b83614e117\") " pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.347480 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.347386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf76r\" (UniqueName: \"kubernetes.io/projected/46a415f3-4b6c-42ef-acce-0b4d7d470631-kube-api-access-sf76r\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:08.439466 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.439435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4"
Sep 17 15:40:08.458101 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.458043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:08.502913 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.502872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj"]
Sep 17 15:40:08.602143 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.602110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9jg52"]
Sep 17 15:40:08.605072 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:40:08.605049    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c959272_5175_4063_bf12_e0b9b8e38650.slice/crio-d8e598334b99cec9ce5256b55df9686dd71ee7d1319789ff65c710d01d1817d9 WatchSource:0}: Error finding container d8e598334b99cec9ce5256b55df9686dd71ee7d1319789ff65c710d01d1817d9: Status 404 returned error can't find the container with id d8e598334b99cec9ce5256b55df9686dd71ee7d1319789ff65c710d01d1817d9
Sep 17 15:40:08.705073 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.705051    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4"]
Sep 17 15:40:08.706989 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:40:08.706957    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7ac916_0662_4e55_860d_da12c759654f.slice/crio-607699175afe3b69a0e3aaf7bd80bbd4e25a39bf49f76f4699db35c8542ea8b2 WatchSource:0}: Error finding container 607699175afe3b69a0e3aaf7bd80bbd4e25a39bf49f76f4699db35c8542ea8b2: Status 404 returned error can't find the container with id 607699175afe3b69a0e3aaf7bd80bbd4e25a39bf49f76f4699db35c8542ea8b2
Sep 17 15:40:08.729822 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.729780    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-k7mhd"]
Sep 17 15:40:08.732153 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:40:08.732115    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2037b600_0a62_4629_93a6_16b83614e117.slice/crio-3dbf28d91d238dddd54f615ea41910246fe64d9f52f4bbdaaa9fe6acc9881e6b WatchSource:0}: Error finding container 3dbf28d91d238dddd54f615ea41910246fe64d9f52f4bbdaaa9fe6acc9881e6b: Status 404 returned error can't find the container with id 3dbf28d91d238dddd54f615ea41910246fe64d9f52f4bbdaaa9fe6acc9881e6b
Sep 17 15:40:08.842620 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:08.842592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:08.842767 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:08.842721    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:08.842859 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:08.842817    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls podName:46a415f3-4b6c-42ef-acce-0b4d7d470631 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:09.842778093 +0000 UTC m=+94.685140890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6dthc" (UID: "46a415f3-4b6c-42ef-acce-0b4d7d470631") : secret "samples-operator-tls" not found
Sep 17 15:40:09.164569 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:09.164488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4" event={"ID":"2b7ac916-0662-4e55-860d-da12c759654f","Type":"ContainerStarted","Data":"607699175afe3b69a0e3aaf7bd80bbd4e25a39bf49f76f4699db35c8542ea8b2"}
Sep 17 15:40:09.168555 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:09.168450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9jg52" event={"ID":"0c959272-5175-4063-bf12-e0b9b8e38650","Type":"ContainerStarted","Data":"d8e598334b99cec9ce5256b55df9686dd71ee7d1319789ff65c710d01d1817d9"}
Sep 17 15:40:09.170332 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:09.170266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" event={"ID":"2037b600-0a62-4629-93a6-16b83614e117","Type":"ContainerStarted","Data":"3dbf28d91d238dddd54f615ea41910246fe64d9f52f4bbdaaa9fe6acc9881e6b"}
Sep 17 15:40:09.171974 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:09.171928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj" event={"ID":"a6762993-a05b-4825-8b15-c77c144e43e3","Type":"ContainerStarted","Data":"fdb8bcfcd3a2b8d6f03a57470317bfa27f846f30a8930d7b35d7e9cdb7af4a80"}
Sep 17 15:40:09.851202 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:09.851082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:09.851403 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:09.851266    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:09.851403 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:09.851354    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls podName:46a415f3-4b6c-42ef-acce-0b4d7d470631 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:11.851328147 +0000 UTC m=+96.693690937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6dthc" (UID: "46a415f3-4b6c-42ef-acce-0b4d7d470631") : secret "samples-operator-tls" not found
Sep 17 15:40:10.176848 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:10.176814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4" event={"ID":"2b7ac916-0662-4e55-860d-da12c759654f","Type":"ContainerStarted","Data":"83b79574b9a5f532d495fdf6eb600e4d81580e9db1d470fa313f73b9c2d8ed63"}
Sep 17 15:40:10.193052 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:10.193002    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-7xgj4" podStartSLOduration=1.000608636 podStartE2EDuration="2.192979835s" totalImagesPullingTime="1.192371199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:10.192079098 +0000 UTC m=+95.034441904" watchObservedRunningTime="2026-09-17 15:40:10.192979835 +0000 UTC m=+95.035342641"
Sep 17 15:40:10.273763 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:10.273734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:10.341654 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:10.341621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:11.747106 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:11.747070    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-fd27s"]
Sep 17 15:40:11.750420 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:11.750400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fd27s"
Sep 17 15:40:11.752301 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:11.752278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pwrqv\""
Sep 17 15:40:11.758773 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:11.757531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-fd27s"]
Sep 17 15:40:11.868310 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:11.868278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:11.868473 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:11.868334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwhm\" (UniqueName: \"kubernetes.io/projected/ab0d93bb-ef0f-42d0-85fe-191c96c69929-kube-api-access-chwhm\") pod \"network-check-source-6f8b768455-fd27s\" (UID: \"ab0d93bb-ef0f-42d0-85fe-191c96c69929\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-fd27s"
Sep 17 15:40:11.868473 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:11.868441    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:11.868583 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:11.868516    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls podName:46a415f3-4b6c-42ef-acce-0b4d7d470631 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:15.868494749 +0000 UTC m=+100.710857532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6dthc" (UID: "46a415f3-4b6c-42ef-acce-0b4d7d470631") : secret "samples-operator-tls" not found
Sep 17 15:40:11.969693 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:11.969615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwhm\" (UniqueName: \"kubernetes.io/projected/ab0d93bb-ef0f-42d0-85fe-191c96c69929-kube-api-access-chwhm\") pod \"network-check-source-6f8b768455-fd27s\" (UID: \"ab0d93bb-ef0f-42d0-85fe-191c96c69929\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-fd27s"
Sep 17 15:40:11.977709 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:11.977688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwhm\" (UniqueName: \"kubernetes.io/projected/ab0d93bb-ef0f-42d0-85fe-191c96c69929-kube-api-access-chwhm\") pod \"network-check-source-6f8b768455-fd27s\" (UID: \"ab0d93bb-ef0f-42d0-85fe-191c96c69929\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-fd27s"
Sep 17 15:40:12.061731 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.061700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fd27s"
Sep 17 15:40:12.076127 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.076089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.185166 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.185136    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/0.log"
Sep 17 15:40:12.185318 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.185185    2580 generic.go:356] "Generic (PLEG): container finished" podID="2037b600-0a62-4629-93a6-16b83614e117" containerID="872558f6d46353790f6dc57fd07786a7cebf1501e3a0e93c091bcd6821a08e3d" exitCode=255
Sep 17 15:40:12.185318 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.185232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" event={"ID":"2037b600-0a62-4629-93a6-16b83614e117","Type":"ContainerDied","Data":"872558f6d46353790f6dc57fd07786a7cebf1501e3a0e93c091bcd6821a08e3d"}
Sep 17 15:40:12.185510 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.185494    2580 scope.go:117] "RemoveContainer" containerID="872558f6d46353790f6dc57fd07786a7cebf1501e3a0e93c091bcd6821a08e3d"
Sep 17 15:40:12.187018 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.186993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj" event={"ID":"a6762993-a05b-4825-8b15-c77c144e43e3","Type":"ContainerStarted","Data":"8cea5469a3fa0ac2028ee751a2030c204a4e917441764daf2e9f186565bbcfba"}
Sep 17 15:40:12.188547 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.188522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9jg52" event={"ID":"0c959272-5175-4063-bf12-e0b9b8e38650","Type":"ContainerStarted","Data":"0f494c4951bf47c7a77b20f8406a508daa374fddd96e99b52864b4384386177a"}
Sep 17 15:40:12.219604 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.219561    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj" podStartSLOduration=2.088703903 podStartE2EDuration="5.219543478s" totalImagesPullingTime="3.130839575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:12.219186382 +0000 UTC m=+97.061549187" watchObservedRunningTime="2026-09-17 15:40:12.219543478 +0000 UTC m=+97.061906285"
Sep 17 15:40:12.234377 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.234327    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-9jg52" podStartSLOduration=1.199321179 podStartE2EDuration="4.23430694s" totalImagesPullingTime="3.034985761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:12.233911294 +0000 UTC m=+97.076274101" watchObservedRunningTime="2026-09-17 15:40:12.23430694 +0000 UTC m=+97.076669748"
Sep 17 15:40:12.342298 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.342264    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.346331 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.346308    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-fd27s"]
Sep 17 15:40:12.349707 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:40:12.349681    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0d93bb_ef0f_42d0_85fe_191c96c69929.slice/crio-e34a3dfd87c841346f1ba97e51c9118a2e6aa6360aa49f472c50272b00907b47 WatchSource:0}: Error finding container e34a3dfd87c841346f1ba97e51c9118a2e6aa6360aa49f472c50272b00907b47: Status 404 returned error can't find the container with id e34a3dfd87c841346f1ba97e51c9118a2e6aa6360aa49f472c50272b00907b47
Sep 17 15:40:12.439711 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.439683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.565539 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.565459    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vfpmt"]
Sep 17 15:40:12.569010 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.568980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.570046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.570020    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.570790 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.570765    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:40:12.570915 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.570769    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9r9ks\""
Sep 17 15:40:12.570915 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.570879    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:40:12.577010 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.576989    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vfpmt"]
Sep 17 15:40:12.632473 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.632450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.675537 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.675508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.675537 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.675537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8v9r\" (UniqueName: \"kubernetes.io/projected/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-api-access-s8v9r\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.675725 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.675602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfec7824-0193-4d02-a151-e149f7bb3d61-crio-socket\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.675725 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.675640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.675725 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.675700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfec7824-0193-4d02-a151-e149f7bb3d61-data-volume\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.777020 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.776989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfec7824-0193-4d02-a151-e149f7bb3d61-data-volume\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.777020 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.777022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.777512 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.777040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8v9r\" (UniqueName: \"kubernetes.io/projected/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-api-access-s8v9r\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.777512 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:12.777129    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:12.777512 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:12.777209    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.27718767 +0000 UTC m=+98.119550469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:12.777512 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.777256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfec7824-0193-4d02-a151-e149f7bb3d61-crio-socket\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.777512 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.777290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.777512 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.777340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfec7824-0193-4d02-a151-e149f7bb3d61-crio-socket\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.777512 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:12.777397    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:12.777512 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.777435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfec7824-0193-4d02-a151-e149f7bb3d61-data-volume\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.777512 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:12.777446    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.277432674 +0000 UTC m=+98.119795456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:12.794709 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.794679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8v9r\" (UniqueName: \"kubernetes.io/projected/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-api-access-s8v9r\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:12.829061 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.828986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:12.877825 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.877775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:40:12.877970 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:12.877902    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:40:12.877970 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:12.877967    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls podName:38dc1beb-42f8-4732-bc67-a5a9f39a97e6 nodeName:}" failed. No retries permitted until 2026-09-17 15:41:16.877951398 +0000 UTC m=+161.720314180 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls") pod "dns-default-bdtrg" (UID: "38dc1beb-42f8-4732-bc67-a5a9f39a97e6") : secret "dns-default-metrics-tls" not found
Sep 17 15:40:12.930652 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:12.930621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:13.193453 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.193423    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/1.log"
Sep 17 15:40:13.193897 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.193882    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/0.log"
Sep 17 15:40:13.193954 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.193919    2580 generic.go:356] "Generic (PLEG): container finished" podID="2037b600-0a62-4629-93a6-16b83614e117" containerID="692b90b650a0fbdeae016fb524b8283edd1b10c590b88a1c9846d2137ac64df5" exitCode=255
Sep 17 15:40:13.193998 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.193955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" event={"ID":"2037b600-0a62-4629-93a6-16b83614e117","Type":"ContainerDied","Data":"692b90b650a0fbdeae016fb524b8283edd1b10c590b88a1c9846d2137ac64df5"}
Sep 17 15:40:13.194046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.194015    2580 scope.go:117] "RemoveContainer" containerID="872558f6d46353790f6dc57fd07786a7cebf1501e3a0e93c091bcd6821a08e3d"
Sep 17 15:40:13.194250 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.194229    2580 scope.go:117] "RemoveContainer" containerID="692b90b650a0fbdeae016fb524b8283edd1b10c590b88a1c9846d2137ac64df5"
Sep 17 15:40:13.194435 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:13.194414    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-k7mhd_openshift-console-operator(2037b600-0a62-4629-93a6-16b83614e117)\"" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" podUID="2037b600-0a62-4629-93a6-16b83614e117"
Sep 17 15:40:13.195754 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.195725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fd27s" event={"ID":"ab0d93bb-ef0f-42d0-85fe-191c96c69929","Type":"ContainerStarted","Data":"c7dbc74b75e67604d890c34475dd8ef1abe1cf88f95607bc9582299fe19875ab"}
Sep 17 15:40:13.195866 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.195775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fd27s" event={"ID":"ab0d93bb-ef0f-42d0-85fe-191c96c69929","Type":"ContainerStarted","Data":"e34a3dfd87c841346f1ba97e51c9118a2e6aa6360aa49f472c50272b00907b47"}
Sep 17 15:40:13.227005 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.226963    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fd27s" podStartSLOduration=2.226950978 podStartE2EDuration="2.226950978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:13.22631605 +0000 UTC m=+98.068678855" watchObservedRunningTime="2026-09-17 15:40:13.226950978 +0000 UTC m=+98.069313783"
Sep 17 15:40:13.279792 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.279766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:13.279946 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:13.279902    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:13.279999 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:13.279961    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:14.279942041 +0000 UTC m=+99.122304840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:13.280133 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.280110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:13.280226 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:13.280212    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:13.280287 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:13.280272    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:14.280253208 +0000 UTC m=+99.122616011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:13.666856 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.666819    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"]
Sep 17 15:40:13.671196 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.671167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.672988 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.672967    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 15:40:13.673092 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.673046    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 15:40:13.673217 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.673191    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 15:40:13.673332 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.673252    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 15:40:13.680376 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.680352    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"]
Sep 17 15:40:13.685550 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.685531    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"]
Sep 17 15:40:13.688545 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.688528    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.690759 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.690737    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 15:40:13.690939 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.690921    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 15:40:13.691028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.690937    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 15:40:13.691028 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.691010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 15:40:13.700784 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.700765    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"]
Sep 17 15:40:13.783477 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.783441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm95q\" (UniqueName: \"kubernetes.io/projected/72b58725-8015-4237-a6ac-575c04a2e3b5-kube-api-access-bm95q\") pod \"klusterlet-addon-workmgr-68987cb78b-zdjn5\" (UID: \"72b58725-8015-4237-a6ac-575c04a2e3b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.783477 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.783474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedadda0-2d4d-4ef4-8066-6b8965b876bc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.783975 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.783552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.783975 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.783575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8qc\" (UniqueName: \"kubernetes.io/projected/fedadda0-2d4d-4ef4-8066-6b8965b876bc-kube-api-access-js8qc\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.783975 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.783603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-ca\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.783975 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.783621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-hub\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.783975 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.783637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/72b58725-8015-4237-a6ac-575c04a2e3b5-klusterlet-config\") pod \"klusterlet-addon-workmgr-68987cb78b-zdjn5\" (UID: \"72b58725-8015-4237-a6ac-575c04a2e3b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.783975 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.783737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72b58725-8015-4237-a6ac-575c04a2e3b5-tmp\") pod \"klusterlet-addon-workmgr-68987cb78b-zdjn5\" (UID: \"72b58725-8015-4237-a6ac-575c04a2e3b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.783975 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.783783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.851129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.851094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:13.884425 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.884400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm95q\" (UniqueName: \"kubernetes.io/projected/72b58725-8015-4237-a6ac-575c04a2e3b5-kube-api-access-bm95q\") pod \"klusterlet-addon-workmgr-68987cb78b-zdjn5\" (UID: \"72b58725-8015-4237-a6ac-575c04a2e3b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.884516 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.884435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedadda0-2d4d-4ef4-8066-6b8965b876bc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.884516 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.884488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.884618 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.884518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8qc\" (UniqueName: \"kubernetes.io/projected/fedadda0-2d4d-4ef4-8066-6b8965b876bc-kube-api-access-js8qc\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.884618 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.884543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-ca\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.884618 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.884566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-hub\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.884618 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.884586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/72b58725-8015-4237-a6ac-575c04a2e3b5-klusterlet-config\") pod \"klusterlet-addon-workmgr-68987cb78b-zdjn5\" (UID: \"72b58725-8015-4237-a6ac-575c04a2e3b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.884918 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.884892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72b58725-8015-4237-a6ac-575c04a2e3b5-tmp\") pod \"klusterlet-addon-workmgr-68987cb78b-zdjn5\" (UID: \"72b58725-8015-4237-a6ac-575c04a2e3b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.885013 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.884943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.885484 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.885422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedadda0-2d4d-4ef4-8066-6b8965b876bc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.885620 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.885559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72b58725-8015-4237-a6ac-575c04a2e3b5-tmp\") pod \"klusterlet-addon-workmgr-68987cb78b-zdjn5\" (UID: \"72b58725-8015-4237-a6ac-575c04a2e3b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.887563 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.887519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.887664 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.887567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-ca\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.887939 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.887920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.888051 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.888032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/72b58725-8015-4237-a6ac-575c04a2e3b5-klusterlet-config\") pod \"klusterlet-addon-workmgr-68987cb78b-zdjn5\" (UID: \"72b58725-8015-4237-a6ac-575c04a2e3b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.888205 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.888189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fedadda0-2d4d-4ef4-8066-6b8965b876bc-hub\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.894898 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.894867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm95q\" (UniqueName: \"kubernetes.io/projected/72b58725-8015-4237-a6ac-575c04a2e3b5-kube-api-access-bm95q\") pod \"klusterlet-addon-workmgr-68987cb78b-zdjn5\" (UID: \"72b58725-8015-4237-a6ac-575c04a2e3b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:13.895124 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.895105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8qc\" (UniqueName: \"kubernetes.io/projected/fedadda0-2d4d-4ef4-8066-6b8965b876bc-kube-api-access-js8qc\") pod \"cluster-proxy-proxy-agent-67bf848c4d-hn8bd\" (UID: \"fedadda0-2d4d-4ef4-8066-6b8965b876bc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:13.908062 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.908030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:13.981949 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:13.981889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:14.006174 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.006147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:14.200772 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.200745    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/1.log"
Sep 17 15:40:14.201109 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.201085    2580 scope.go:117] "RemoveContainer" containerID="692b90b650a0fbdeae016fb524b8283edd1b10c590b88a1c9846d2137ac64df5"
Sep 17 15:40:14.201238 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:14.201221    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-k7mhd_openshift-console-operator(2037b600-0a62-4629-93a6-16b83614e117)\"" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" podUID="2037b600-0a62-4629-93a6-16b83614e117"
Sep 17 15:40:14.254268 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.253899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"]
Sep 17 15:40:14.256583 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:40:14.256558    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b58725_8015_4237_a6ac_575c04a2e3b5.slice/crio-7e5c382d2d86823f5684cfe70272af508049e334288b3080e2331d8970e23580 WatchSource:0}: Error finding container 7e5c382d2d86823f5684cfe70272af508049e334288b3080e2331d8970e23580: Status 404 returned error can't find the container with id 7e5c382d2d86823f5684cfe70272af508049e334288b3080e2331d8970e23580
Sep 17 15:40:14.291888 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.291848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:14.292052 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:14.291961    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:14.292052 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:14.292032    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:16.29201186 +0000 UTC m=+101.134374663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:14.292191 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.292099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:14.292309 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:14.292285    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:14.292434 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:14.292402    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:16.29232347 +0000 UTC m=+101.134686252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:14.293475 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.293455    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"]
Sep 17 15:40:14.296240 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:40:14.296223    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfedadda0_2d4d_4ef4_8066_6b8965b876bc.slice/crio-311eac8e4df4caff81d25cad17c2bd56354f2edecbf8de170445011ddc3c0bbe WatchSource:0}: Error finding container 311eac8e4df4caff81d25cad17c2bd56354f2edecbf8de170445011ddc3c0bbe: Status 404 returned error can't find the container with id 311eac8e4df4caff81d25cad17c2bd56354f2edecbf8de170445011ddc3c0bbe
Sep 17 15:40:14.485415 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.485385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:14.518636 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.518579    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hv6gp_6ecaf429-99cd-48a9-8eaf-96cdc04dd17b/dns-node-resolver/0.log"
Sep 17 15:40:14.587301 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:14.587279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:15.209865 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:15.209822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5" event={"ID":"72b58725-8015-4237-a6ac-575c04a2e3b5","Type":"ContainerStarted","Data":"7e5c382d2d86823f5684cfe70272af508049e334288b3080e2331d8970e23580"}
Sep 17 15:40:15.211813 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:15.211769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" event={"ID":"fedadda0-2d4d-4ef4-8066-6b8965b876bc","Type":"ContainerStarted","Data":"311eac8e4df4caff81d25cad17c2bd56354f2edecbf8de170445011ddc3c0bbe"}
Sep 17 15:40:15.394557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:15.394522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:15.719600 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:15.719487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rwm2w_62b44d6a-61ff-458c-917c-0f4a105a381d/node-ca/0.log"
Sep 17 15:40:15.905497 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:15.904829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:15.905497 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:15.905064    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:15.905497 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:15.905130    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls podName:46a415f3-4b6c-42ef-acce-0b4d7d470631 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:23.905111513 +0000 UTC m=+108.747474297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6dthc" (UID: "46a415f3-4b6c-42ef-acce-0b4d7d470631") : secret "samples-operator-tls" not found
Sep 17 15:40:16.308563 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:16.308520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:16.309064 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:16.308589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:16.309064 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:16.308726    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:16.309064 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:16.308816    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:20.308779462 +0000 UTC m=+105.151142252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:16.309220 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:16.309193    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:16.309298 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:16.309243    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:20.309231727 +0000 UTC m=+105.151594523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:16.518060 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:16.518031    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zzkfw_08c92ab5-a08c-4d43-9cb6-d20f69be2a2c/serve-healthcheck-canary/0.log"
Sep 17 15:40:17.106182 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:17.106145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jw9rz"
Sep 17 15:40:17.191769 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:17.191734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:17.334085 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:17.334051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:17.477495 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:17.477450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:17.579823 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:17.579764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:17.683565 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:17.683535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:17.777125 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:17.777048    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:17.897953 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:17.897914    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:18.297502 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:18.297468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:18.458895 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:18.458858    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:18.459309 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:18.458932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:18.459394 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:18.459369    2580 scope.go:117] "RemoveContainer" containerID="692b90b650a0fbdeae016fb524b8283edd1b10c590b88a1c9846d2137ac64df5"
Sep 17 15:40:18.459609 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:18.459580    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-k7mhd_openshift-console-operator(2037b600-0a62-4629-93a6-16b83614e117)\"" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" podUID="2037b600-0a62-4629-93a6-16b83614e117"
Sep 17 15:40:18.887242 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:18.887211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:19.287015 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:19.286939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:19.482708 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:19.482672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:19.681204 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:19.681159    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:20.088776 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:20.088706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:20.344045 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:20.343970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:20.344045 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:20.344019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:20.344219 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:20.344116    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:20.344219 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:20.344139    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:20.344219 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:20.344176    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:28.344163127 +0000 UTC m=+113.186525910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:20.344219 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:20.344202    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls podName:bfec7824-0193-4d02-a151-e149f7bb3d61 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:28.344187403 +0000 UTC m=+113.186550185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfpmt" (UID: "bfec7824-0193-4d02-a151-e149f7bb3d61") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:20.685528 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:20.685506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:20.881619 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:20.881577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:20.987791 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:20.987756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:21.088230 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.088199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:21.236116 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.236031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" event={"ID":"fedadda0-2d4d-4ef4-8066-6b8965b876bc","Type":"ContainerStarted","Data":"bd083e920cff993fbd69753d6e6b8f6aa5ea53d5d18446f364e387c9ed40669e"}
Sep 17 15:40:21.236116 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.236080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" event={"ID":"fedadda0-2d4d-4ef4-8066-6b8965b876bc","Type":"ContainerStarted","Data":"1f8f4f057d66c727b662ac3eeb93c0170637e5a5221bee59c373729c5ee8a73e"}
Sep 17 15:40:21.236116 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.236101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" event={"ID":"fedadda0-2d4d-4ef4-8066-6b8965b876bc","Type":"ContainerStarted","Data":"d185af91fb027f502150a9c69702d7faff79399bd2d039ad353d42fb8f725762"}
Sep 17 15:40:21.237477 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.237456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5" event={"ID":"72b58725-8015-4237-a6ac-575c04a2e3b5","Type":"ContainerStarted","Data":"ef59e81684c6ee27682657bc54817ffdbc6876f25b6f077aacc1d76b623da277"}
Sep 17 15:40:21.237651 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.237641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:21.239199 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.239181    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5"
Sep 17 15:40:21.255167 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.255126    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" podStartSLOduration=2.152993962 podStartE2EDuration="8.255113187s" totalImagesPullingTime="6.102119225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:21.254390267 +0000 UTC m=+106.096753072" watchObservedRunningTime="2026-09-17 15:40:21.255113187 +0000 UTC m=+106.097476022"
Sep 17 15:40:21.269651 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.269614    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68987cb78b-zdjn5" podStartSLOduration=2.114599982 podStartE2EDuration="8.269602484s" totalImagesPullingTime="6.155002502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:21.269598561 +0000 UTC m=+106.111961365" watchObservedRunningTime="2026-09-17 15:40:21.269602484 +0000 UTC m=+106.111965289"
Sep 17 15:40:21.490866 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.490768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:21.795404 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.795314    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:21.896002 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.895977    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:21.996690 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:21.996646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:22.287008 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:22.286979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:22.484950 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:22.484920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:22.689702 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:22.689674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:23.384146 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:23.384112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:23.683283 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:23.683196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:23.972649 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:23.972571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:23.975058 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:23.975030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/46a415f3-4b6c-42ef-acce-0b4d7d470631-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6dthc\" (UID: \"46a415f3-4b6c-42ef-acce-0b4d7d470631\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:24.008659 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:24.008622    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" podUID="fedadda0-2d4d-4ef4-8066-6b8965b876bc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 15:40:24.062730 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:24.062703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"
Sep 17 15:40:24.123722 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:24.123693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:24.286123 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:24.286052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:24.327000 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:24.326879    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc"]
Sep 17 15:40:24.486182 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:24.486147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:24.685633 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:24.685599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:25.252708 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:25.252670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc" event={"ID":"46a415f3-4b6c-42ef-acce-0b4d7d470631","Type":"ContainerStarted","Data":"5e1f021a7244959e4f34ba47dd3e2c417ece7d1c7e0fffe26b1e582908b62b96"}
Sep 17 15:40:25.289056 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:25.289027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:25.549502 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:25.549406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:25.894574 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:25.894531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:26.258041 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:26.258011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc" event={"ID":"46a415f3-4b6c-42ef-acce-0b4d7d470631","Type":"ContainerStarted","Data":"3dddf95644b0b01622bf95d58449aa5358e3be56ba7112c7decdb52846362732"}
Sep 17 15:40:26.258209 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:26.258063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc" event={"ID":"46a415f3-4b6c-42ef-acce-0b4d7d470631","Type":"ContainerStarted","Data":"08428fa3e4c937bce76d8c0944b77c44d0ba89a5bbc4eda184d4308f443fc9ca"}
Sep 17 15:40:26.273240 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:26.273201    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6dthc" podStartSLOduration=16.568698138 podStartE2EDuration="18.273186523s" totalImagesPullingTime="1.704488385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:26.272766576 +0000 UTC m=+111.115129383" watchObservedRunningTime="2026-09-17 15:40:26.273186523 +0000 UTC m=+111.115549328"
Sep 17 15:40:26.354606 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:26.354577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:26.486389 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:26.486312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:26.884835 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:26.884792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:27.489225 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:27.489195    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:27.682034 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:27.682005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:27.892265 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:27.892236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:28.118731 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.118702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:28.222719 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.222638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:28.319633 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.319583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:28.407978 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.407948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:28.408129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.408045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:28.408572 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.408555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfec7824-0193-4d02-a151-e149f7bb3d61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:28.410489 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.410467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfec7824-0193-4d02-a151-e149f7bb3d61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfpmt\" (UID: \"bfec7824-0193-4d02-a151-e149f7bb3d61\") " pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:28.420560 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.420540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:28.478494 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.478439    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vfpmt"
Sep 17 15:40:28.746538 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:28.746515    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vfpmt"]
Sep 17 15:40:28.749006 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:40:28.748964    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfec7824_0193_4d02_a151_e149f7bb3d61.slice/crio-fb9d16a2fee608eeeea31ebda3c9c8f9d6d850ce0572851a43240b413e2c79f1 WatchSource:0}: Error finding container fb9d16a2fee608eeeea31ebda3c9c8f9d6d850ce0572851a43240b413e2c79f1: Status 404 returned error can't find the container with id fb9d16a2fee608eeeea31ebda3c9c8f9d6d850ce0572851a43240b413e2c79f1
Sep 17 15:40:29.268991 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:29.268955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfpmt" event={"ID":"bfec7824-0193-4d02-a151-e149f7bb3d61","Type":"ContainerStarted","Data":"027cb7b1e115afc52b8dd3d259a653bd9ea514002f945adaa60df1049586917d"}
Sep 17 15:40:29.269319 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:29.269012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfpmt" event={"ID":"bfec7824-0193-4d02-a151-e149f7bb3d61","Type":"ContainerStarted","Data":"fb9d16a2fee608eeeea31ebda3c9c8f9d6d850ce0572851a43240b413e2c79f1"}
Sep 17 15:40:30.274369 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:30.274337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfpmt" event={"ID":"bfec7824-0193-4d02-a151-e149f7bb3d61","Type":"ContainerStarted","Data":"e5d95ead4f57de4a5e7b20fdc4e02ab78a26b762afc92b6673e4aa5708f2ec5c"}
Sep 17 15:40:30.669538 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:30.669510    2580 scope.go:117] "RemoveContainer" containerID="692b90b650a0fbdeae016fb524b8283edd1b10c590b88a1c9846d2137ac64df5"
Sep 17 15:40:31.066019 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:31.065916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:31.132002 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:31.131966    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:31.279834 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:31.279793    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:40:31.280244 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:31.280227    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/1.log"
Sep 17 15:40:31.280283 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:31.280269    2580 generic.go:356] "Generic (PLEG): container finished" podID="2037b600-0a62-4629-93a6-16b83614e117" containerID="b67772c5bc33b33554c94641fd5a79d74c09998e6f688718c05e5b5777b0332e" exitCode=255
Sep 17 15:40:31.280362 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:31.280338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" event={"ID":"2037b600-0a62-4629-93a6-16b83614e117","Type":"ContainerDied","Data":"b67772c5bc33b33554c94641fd5a79d74c09998e6f688718c05e5b5777b0332e"}
Sep 17 15:40:31.281082 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:31.280394    2580 scope.go:117] "RemoveContainer" containerID="692b90b650a0fbdeae016fb524b8283edd1b10c590b88a1c9846d2137ac64df5"
Sep 17 15:40:31.281184 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:31.281152    2580 scope.go:117] "RemoveContainer" containerID="b67772c5bc33b33554c94641fd5a79d74c09998e6f688718c05e5b5777b0332e"
Sep 17 15:40:31.281356 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:31.281336    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-k7mhd_openshift-console-operator(2037b600-0a62-4629-93a6-16b83614e117)\"" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" podUID="2037b600-0a62-4629-93a6-16b83614e117"
Sep 17 15:40:32.291214 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:32.291178    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfpmt" event={"ID":"bfec7824-0193-4d02-a151-e149f7bb3d61","Type":"ContainerStarted","Data":"0049de69efb913d7c33b0749ca7950be141df993c144e573749fb7df5f873d26"}
Sep 17 15:40:32.292734 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:32.292716    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:40:32.312855 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:32.312790    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vfpmt" podStartSLOduration=17.94182318 podStartE2EDuration="20.312777961s" totalImagesPullingTime="2.370954781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:32.312366898 +0000 UTC m=+117.154729702" watchObservedRunningTime="2026-09-17 15:40:32.312777961 +0000 UTC m=+117.155140765"
Sep 17 15:40:34.008070 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:34.008031    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" podUID="fedadda0-2d4d-4ef4-8066-6b8965b876bc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 15:40:34.194975 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:34.194932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:34.287372 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:34.287296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:34.386552 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:34.386522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:36.337618 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:36.337582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:36.414673 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:36.414633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:36.470981 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:36.470949    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:36.531610 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:36.531582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:38.454356 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:38.454326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:38.458313 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:38.458292    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:38.458427 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:38.458351    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:38.458710 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:38.458695    2580 scope.go:117] "RemoveContainer" containerID="b67772c5bc33b33554c94641fd5a79d74c09998e6f688718c05e5b5777b0332e"
Sep 17 15:40:38.458917 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:38.458897    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-k7mhd_openshift-console-operator(2037b600-0a62-4629-93a6-16b83614e117)\"" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" podUID="2037b600-0a62-4629-93a6-16b83614e117"
Sep 17 15:40:40.503976 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.503948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.613757 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.613723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.742877 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.742836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.785842 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.785732    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vsbdp"]
Sep 17 15:40:40.790171 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.790152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.792557 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.792533    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:40:40.792696 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.792533    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7zh5h\""
Sep 17 15:40:40.792696 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.792536    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:40:40.793035 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.793020    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:40:40.793165 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.793149    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:40:40.793241 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.793194    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:40:40.793305 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.793242    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:40:40.837935 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.837910    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.897755 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.897722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.897919 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.897763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7015c765-eb52-4355-a95c-54bc96a4ac9a-metrics-client-ca\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.897919 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.897841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-textfile\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.897919 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.897863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-wtmp\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.898023 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.897921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.898023 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.897954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfdks\" (UniqueName: \"kubernetes.io/projected/7015c765-eb52-4355-a95c-54bc96a4ac9a-kube-api-access-wfdks\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.898023 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.897975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7015c765-eb52-4355-a95c-54bc96a4ac9a-root\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.898127 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.898032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7015c765-eb52-4355-a95c-54bc96a4ac9a-sys\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.898127 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.898047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-tls\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.904929 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.904904    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:40.999043 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-wtmp\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999202 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999202 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfdks\" (UniqueName: \"kubernetes.io/projected/7015c765-eb52-4355-a95c-54bc96a4ac9a-kube-api-access-wfdks\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999202 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7015c765-eb52-4355-a95c-54bc96a4ac9a-root\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999365 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-wtmp\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999365 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7015c765-eb52-4355-a95c-54bc96a4ac9a-root\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999476 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7015c765-eb52-4355-a95c-54bc96a4ac9a-sys\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999476 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-tls\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7015c765-eb52-4355-a95c-54bc96a4ac9a-sys\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7015c765-eb52-4355-a95c-54bc96a4ac9a-metrics-client-ca\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999708 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:40.999619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-textfile\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:40.999708 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:40.999650    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 15:40:40.999789 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:40:40.999727    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-tls podName:7015c765-eb52-4355-a95c-54bc96a4ac9a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:41.499705029 +0000 UTC m=+126.342067814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-tls") pod "node-exporter-vsbdp" (UID: "7015c765-eb52-4355-a95c-54bc96a4ac9a") : secret "node-exporter-tls" not found
Sep 17 15:40:41.000052 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.000025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-textfile\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:41.000194 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.000158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:41.000322 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.000301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7015c765-eb52-4355-a95c-54bc96a4ac9a-metrics-client-ca\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:41.001956 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.001897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:41.014186 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.014164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfdks\" (UniqueName: \"kubernetes.io/projected/7015c765-eb52-4355-a95c-54bc96a4ac9a-kube-api-access-wfdks\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:41.047722 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.047636    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.148076 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.148045    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.247842 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.247817    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.349651 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.349584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.504135 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.504104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-tls\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:41.506505 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.506481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7015c765-eb52-4355-a95c-54bc96a4ac9a-node-exporter-tls\") pod \"node-exporter-vsbdp\" (UID: \"7015c765-eb52-4355-a95c-54bc96a4ac9a\") " pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:41.569776 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.569747    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.700564 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.700537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vsbdp"
Sep 17 15:40:41.711410 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:40:41.711382    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7015c765_eb52_4355_a95c_54bc96a4ac9a.slice/crio-c8372f220ff85d2fb565964f2c123f772798c1bd9d3902699959d0bf61d64d91 WatchSource:0}: Error finding container c8372f220ff85d2fb565964f2c123f772798c1bd9d3902699959d0bf61d64d91: Status 404 returned error can't find the container with id c8372f220ff85d2fb565964f2c123f772798c1bd9d3902699959d0bf61d64d91
Sep 17 15:40:41.746981 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.746935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:41.823405 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:41.823364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:42.189981 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:42.189940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:42.328268 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:42.328238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vsbdp" event={"ID":"7015c765-eb52-4355-a95c-54bc96a4ac9a","Type":"ContainerStarted","Data":"c8372f220ff85d2fb565964f2c123f772798c1bd9d3902699959d0bf61d64d91"}
Sep 17 15:40:43.333090 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:43.333056    2580 generic.go:356] "Generic (PLEG): container finished" podID="7015c765-eb52-4355-a95c-54bc96a4ac9a" containerID="2aee8b9708d903451c97c54e576a49661d1e134450ec75141e084488d9a45d73" exitCode=0
Sep 17 15:40:43.333465 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:43.333122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vsbdp" event={"ID":"7015c765-eb52-4355-a95c-54bc96a4ac9a","Type":"ContainerDied","Data":"2aee8b9708d903451c97c54e576a49661d1e134450ec75141e084488d9a45d73"}
Sep 17 15:40:43.370566 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:43.370544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:43.745511 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:43.745477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:43.832780 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:43.832750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:44.007513 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.007437    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" podUID="fedadda0-2d4d-4ef4-8066-6b8965b876bc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 15:40:44.007513 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.007501    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd"
Sep 17 15:40:44.008083 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.008051    2580 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"bd083e920cff993fbd69753d6e6b8f6aa5ea53d5d18446f364e387c9ed40669e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 15:40:44.008129 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.008109    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" podUID="fedadda0-2d4d-4ef4-8066-6b8965b876bc" containerName="service-proxy" containerID="cri-o://bd083e920cff993fbd69753d6e6b8f6aa5ea53d5d18446f364e387c9ed40669e" gracePeriod=30
Sep 17 15:40:44.019487 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.019464    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:44.339264 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.339194    2580 generic.go:356] "Generic (PLEG): container finished" podID="fedadda0-2d4d-4ef4-8066-6b8965b876bc" containerID="bd083e920cff993fbd69753d6e6b8f6aa5ea53d5d18446f364e387c9ed40669e" exitCode=2
Sep 17 15:40:44.339639 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.339273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" event={"ID":"fedadda0-2d4d-4ef4-8066-6b8965b876bc","Type":"ContainerDied","Data":"bd083e920cff993fbd69753d6e6b8f6aa5ea53d5d18446f364e387c9ed40669e"}
Sep 17 15:40:44.339639 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.339320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67bf848c4d-hn8bd" event={"ID":"fedadda0-2d4d-4ef4-8066-6b8965b876bc","Type":"ContainerStarted","Data":"4eae867727b031cfd1f603ae7494bb187a51fc9397435948e902ff1c759e27ce"}
Sep 17 15:40:44.341426 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.341406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vsbdp" event={"ID":"7015c765-eb52-4355-a95c-54bc96a4ac9a","Type":"ContainerStarted","Data":"28e6c37698320008e153ac0f9a8987bdcb4ef4863cb119fe1cee11392dd246d0"}
Sep 17 15:40:44.341530 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.341444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vsbdp" event={"ID":"7015c765-eb52-4355-a95c-54bc96a4ac9a","Type":"ContainerStarted","Data":"43414671cab64580f092c1a47d6e7c0ab04e0a0b297e011833c8032ab5918f5f"}
Sep 17 15:40:44.375272 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.375244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:44.386359 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.386317    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vsbdp" podStartSLOduration=3.472445082 podStartE2EDuration="4.386303477s" totalImagesPullingTime="840.754059ms" totalInitContainerRuntime="73.104336ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:44.38560303 +0000 UTC m=+129.227965827" watchObservedRunningTime="2026-09-17 15:40:44.386303477 +0000 UTC m=+129.228666673"
Sep 17 15:40:44.529490 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.529451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:40:44.531878 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.531855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86c0b23-dea0-497e-a3ca-fa1037439372-metrics-certs\") pod \"network-metrics-daemon-xztjr\" (UID: \"d86c0b23-dea0-497e-a3ca-fa1037439372\") " pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:40:44.589870 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.589779    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h48gq\""
Sep 17 15:40:44.598550 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.598532    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xztjr"
Sep 17 15:40:44.864285 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:44.864259    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xztjr"]
Sep 17 15:40:44.867262 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:40:44.867238    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86c0b23_dea0_497e_a3ca_fa1037439372.slice/crio-40802f820076a7bd9766b25c1978ccf196a36be974212c14463195b04d822ab0 WatchSource:0}: Error finding container 40802f820076a7bd9766b25c1978ccf196a36be974212c14463195b04d822ab0: Status 404 returned error can't find the container with id 40802f820076a7bd9766b25c1978ccf196a36be974212c14463195b04d822ab0
Sep 17 15:40:45.346818 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:45.346713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xztjr" event={"ID":"d86c0b23-dea0-497e-a3ca-fa1037439372","Type":"ContainerStarted","Data":"40802f820076a7bd9766b25c1978ccf196a36be974212c14463195b04d822ab0"}
Sep 17 15:40:45.539705 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:45.539663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:45.958018 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:45.957987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:46.084475 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:46.084403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:46.150712 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:46.150679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:46.351469 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:46.351385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xztjr" event={"ID":"d86c0b23-dea0-497e-a3ca-fa1037439372","Type":"ContainerStarted","Data":"8eec44fe6f92cc608707657fc58689cb3f4c48ac388f48c1353d4b34cc3c1b1f"}
Sep 17 15:40:46.351469 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:46.351446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xztjr" event={"ID":"d86c0b23-dea0-497e-a3ca-fa1037439372","Type":"ContainerStarted","Data":"0d16068fe8b313a4cc573cd9a2228fd10b5d7e3b9055ed50b2b8231ca98d00e4"}
Sep 17 15:40:46.370001 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:46.369954    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xztjr" podStartSLOduration=130.515960512 podStartE2EDuration="2m11.369940829s" totalImagesPullingTime="853.980317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:46.369242376 +0000 UTC m=+131.211605181" watchObservedRunningTime="2026-09-17 15:40:46.369940829 +0000 UTC m=+131.212303633"
Sep 17 15:40:46.472989 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:46.472949    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:46.573820 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:46.573773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:46.999768 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:46.999738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:49.485397 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:49.485364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:49.539569 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:49.539541    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:51.670072 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:51.670041    2580 scope.go:117] "RemoveContainer" containerID="b67772c5bc33b33554c94641fd5a79d74c09998e6f688718c05e5b5777b0332e"
Sep 17 15:40:52.372838 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:52.372788    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:40:52.372993 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:52.372865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" event={"ID":"2037b600-0a62-4629-93a6-16b83614e117","Type":"ContainerStarted","Data":"dd578958239a5471a5851414c7d20535e8757a41633f6e18c880e8eaefcf22ee"}
Sep 17 15:40:52.373153 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:52.373134    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:52.377841 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:52.377790    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd"
Sep 17 15:40:52.390936 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:52.390885    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-k7mhd" podStartSLOduration=41.478241195 podStartE2EDuration="44.39086828s" totalImagesPullingTime="2.912627085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:52.388734093 +0000 UTC m=+137.231096902" watchObservedRunningTime="2026-09-17 15:40:52.39086828 +0000 UTC m=+137.233231084"
Sep 17 15:40:52.458020 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:52.457984    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:52.556655 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:52.556613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:52.655599 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:52.655505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:52.946166 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:52.946082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.121446 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:53.121390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.222656 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:53.222566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.323272 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:53.323238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.425549 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:53.425524    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.524497 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:53.524406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.625793 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:53.625733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:53.939821 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:53.939763    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:54.145570 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:54.145540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:54.621205 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:54.621176    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:55.236531 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:55.236498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:55.436936 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:55.436904    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:56.403872 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:56.403836    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:56.505833 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:56.505766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:56.607072 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:56.607003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:58.055609 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:58.055540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:58.117552 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:58.117523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:58.567844 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:58.567783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:40:58.671960 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:40:58.671923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:01.064118 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:01.064083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:01.125693 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:01.125639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:07.564561 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:07.564529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:07.622060 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:07.622030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:11.354935 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:11.354878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:12.054539 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:41:12.054504    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bdtrg" podUID="38dc1beb-42f8-4732-bc67-a5a9f39a97e6"
Sep 17 15:41:12.443373 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:12.443341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:41:16.060274 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:16.060245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:16.162815 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:16.162761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:16.886526 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:16.886477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:41:16.888990 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:16.888956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38dc1beb-42f8-4732-bc67-a5a9f39a97e6-metrics-tls\") pod \"dns-default-bdtrg\" (UID: \"38dc1beb-42f8-4732-bc67-a5a9f39a97e6\") " pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:41:16.946752 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:16.946720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghpsh\""
Sep 17 15:41:16.955530 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:16.955505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:41:17.222195 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:17.222107    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bdtrg"]
Sep 17 15:41:17.225106 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:41:17.225080    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dc1beb_42f8_4732_bc67_a5a9f39a97e6.slice/crio-9cab25fdc5f32220b9c50681f70d644ec3945201774e8e8dd05d06fea9fd3413 WatchSource:0}: Error finding container 9cab25fdc5f32220b9c50681f70d644ec3945201774e8e8dd05d06fea9fd3413: Status 404 returned error can't find the container with id 9cab25fdc5f32220b9c50681f70d644ec3945201774e8e8dd05d06fea9fd3413
Sep 17 15:41:17.459950 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:17.459919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdtrg" event={"ID":"38dc1beb-42f8-4732-bc67-a5a9f39a97e6","Type":"ContainerStarted","Data":"9cab25fdc5f32220b9c50681f70d644ec3945201774e8e8dd05d06fea9fd3413"}
Sep 17 15:41:19.366996 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:19.366959    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:19.427142 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:19.427108    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:19.468172 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:19.468140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdtrg" event={"ID":"38dc1beb-42f8-4732-bc67-a5a9f39a97e6","Type":"ContainerStarted","Data":"4a2e3b2c7af91c20c5f4aa7019c13901038dd93dea5d7ce76cde839696b19b87"}
Sep 17 15:41:19.468302 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:19.468197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdtrg" event={"ID":"38dc1beb-42f8-4732-bc67-a5a9f39a97e6","Type":"ContainerStarted","Data":"a37725e9b5292af53e137ddb436990993b42b4972f046897f7db874f55d9330e"}
Sep 17 15:41:19.468302 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:19.468251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:41:19.484623 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:19.484576    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bdtrg" podStartSLOduration=130.048167892 podStartE2EDuration="2m11.484556958s" totalImagesPullingTime="1.436389066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:19.484182831 +0000 UTC m=+164.326545649" watchObservedRunningTime="2026-09-17 15:41:19.484556958 +0000 UTC m=+164.326919761"
Sep 17 15:41:22.482532 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:22.482503    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c959272-5175-4063-bf12-e0b9b8e38650" containerID="0f494c4951bf47c7a77b20f8406a508daa374fddd96e99b52864b4384386177a" exitCode=0
Sep 17 15:41:22.482944 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:22.482546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9jg52" event={"ID":"0c959272-5175-4063-bf12-e0b9b8e38650","Type":"ContainerDied","Data":"0f494c4951bf47c7a77b20f8406a508daa374fddd96e99b52864b4384386177a"}
Sep 17 15:41:22.482944 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:22.482864    2580 scope.go:117] "RemoveContainer" containerID="0f494c4951bf47c7a77b20f8406a508daa374fddd96e99b52864b4384386177a"
Sep 17 15:41:23.401659 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:23.401624    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdtrg_38dc1beb-42f8-4732-bc67-a5a9f39a97e6/dns/0.log"
Sep 17 15:41:23.487538 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:23.487497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9jg52" event={"ID":"0c959272-5175-4063-bf12-e0b9b8e38650","Type":"ContainerStarted","Data":"cf98ea7dd33e0aa22b0c7cdcb3e5c39dea72721ca1bb9442da5ce6986043b5a0"}
Sep 17 15:41:23.604046 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:23.604020    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdtrg_38dc1beb-42f8-4732-bc67-a5a9f39a97e6/kube-rbac-proxy/0.log"
Sep 17 15:41:24.769627 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:24.769588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:24.801783 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:24.801761    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hv6gp_6ecaf429-99cd-48a9-8eaf-96cdc04dd17b/dns-node-resolver/0.log"
Sep 17 15:41:24.870723 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:24.870685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:24.934541 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:24.934512    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:25.800978 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:25.800946    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zzkfw_08c92ab5-a08c-4d43-9cb6-d20f69be2a2c/serve-healthcheck-canary/0.log"
Sep 17 15:41:27.502966 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:27.502932    2580 generic.go:356] "Generic (PLEG): container finished" podID="a6762993-a05b-4825-8b15-c77c144e43e3" containerID="8cea5469a3fa0ac2028ee751a2030c204a4e917441764daf2e9f186565bbcfba" exitCode=0
Sep 17 15:41:27.503427 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:27.502990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj" event={"ID":"a6762993-a05b-4825-8b15-c77c144e43e3","Type":"ContainerDied","Data":"8cea5469a3fa0ac2028ee751a2030c204a4e917441764daf2e9f186565bbcfba"}
Sep 17 15:41:27.503427 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:27.503337    2580 scope.go:117] "RemoveContainer" containerID="8cea5469a3fa0ac2028ee751a2030c204a4e917441764daf2e9f186565bbcfba"
Sep 17 15:41:27.628101 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:27.628067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:27.705858 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:27.705828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:28.512241 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:28.512211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rl5dj" event={"ID":"a6762993-a05b-4825-8b15-c77c144e43e3","Type":"ContainerStarted","Data":"6b3e453646ab2befbc64a910bc9a45e289ae8fc0201f0faa315e63f211ad8796"}
Sep 17 15:41:29.476187 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:29.476156    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bdtrg"
Sep 17 15:41:31.066006 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:31.065978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:31.125755 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:31.125728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:31.279836 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:31.279786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:31.336725 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:31.336648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:40.288705 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:40.288663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:40.351330 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:40.351279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:56.871444 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:56.871414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:41:56.928928 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:41:56.928896    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:00.047085 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:00.047052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:00.101629 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:00.101587    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:02.786352 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:02.786311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:02.837884 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:02.837840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:13.857120 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:13.857082    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:13.962643 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:13.962582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:14.064553 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:14.064517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:16.070168 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:16.070134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:16.165897 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:16.165867    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:16.267330 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:16.267299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:19.609767 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:19.609736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:39.843923 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:39.843889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:42:53.525297 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:42:53.525266    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:06.822474 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:43:06.822395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:06.973485 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:43:06.973452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:43:35.579337 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:43:35.579310    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:43:35.580753 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:43:35.580734    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:43:35.590889 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:43:35.590871    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:44:16.489473 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:44:16.489441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:44:47.454497 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:44:47.454458    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:44:48.763534 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:44:48.763494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:44:49.408624 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:44:49.408588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:44:49.483639 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:44:49.483612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:44:59.995155 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:44:59.995109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:00.093334 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:00.093291    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:00.196655 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:00.196625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:15.267900 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:15.267864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:22.442477 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:22.442438    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:22.538684 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:22.538637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:27.222580 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:27.222550    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:27.541008 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:27.540932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:27.694201 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:27.694173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:28.290842 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.290780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:28.407785 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.407751    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:28.505700 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.505660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:28.505931 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.505911    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-849qb"]
Sep 17 15:45:28.508904 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.508889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:28.510897 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.510873    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 15:45:28.511004 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.510986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 15:45:28.511064 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.510986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-9htvw\""
Sep 17 15:45:28.511342 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.511326    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 15:45:28.511499 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.511361    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 15:45:28.516815 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.516774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-849qb"]
Sep 17 15:45:28.587445 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.587369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-certificates\") pod \"keda-admission-6c4d864b86-849qb\" (UID: \"87c88754-30df-4c85-94e0-1ce0f421e3c1\") " pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:28.587445 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.587405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rhh\" (UniqueName: \"kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-kube-api-access-27rhh\") pod \"keda-admission-6c4d864b86-849qb\" (UID: \"87c88754-30df-4c85-94e0-1ce0f421e3c1\") " pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:28.618959 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.618935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:28.688661 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.688633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-certificates\") pod \"keda-admission-6c4d864b86-849qb\" (UID: \"87c88754-30df-4c85-94e0-1ce0f421e3c1\") " pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:28.688852 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.688675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rhh\" (UniqueName: \"kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-kube-api-access-27rhh\") pod \"keda-admission-6c4d864b86-849qb\" (UID: \"87c88754-30df-4c85-94e0-1ce0f421e3c1\") " pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:28.688852 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:45:28.688819    2580 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 15:45:28.688852 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:45:28.688846    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-849qb: secret "keda-admission-webhooks-certs" not found
Sep 17 15:45:28.689026 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:45:28.688922    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-certificates podName:87c88754-30df-4c85-94e0-1ce0f421e3c1 nodeName:}" failed. No retries permitted until 2026-09-17 15:45:29.188899555 +0000 UTC m=+414.031262339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-certificates") pod "keda-admission-6c4d864b86-849qb" (UID: "87c88754-30df-4c85-94e0-1ce0f421e3c1") : secret "keda-admission-webhooks-certs" not found
Sep 17 15:45:28.698067 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.698047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rhh\" (UniqueName: \"kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-kube-api-access-27rhh\") pod \"keda-admission-6c4d864b86-849qb\" (UID: \"87c88754-30df-4c85-94e0-1ce0f421e3c1\") " pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:28.862571 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:28.862532    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.079552 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:29.079522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.192069 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:29.191991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-certificates\") pod \"keda-admission-6c4d864b86-849qb\" (UID: \"87c88754-30df-4c85-94e0-1ce0f421e3c1\") " pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:29.192206 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:45:29.192135    2580 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 15:45:29.192206 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:45:29.192160    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-849qb: secret "keda-admission-webhooks-certs" not found
Sep 17 15:45:29.192271 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:45:29.192236    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-certificates podName:87c88754-30df-4c85-94e0-1ce0f421e3c1 nodeName:}" failed. No retries permitted until 2026-09-17 15:45:30.192218987 +0000 UTC m=+415.034581770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-certificates") pod "keda-admission-6c4d864b86-849qb" (UID: "87c88754-30df-4c85-94e0-1ce0f421e3c1") : secret "keda-admission-webhooks-certs" not found
Sep 17 15:45:29.231642 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:29.231617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:29.331414 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:29.331387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:45:30.198625 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:30.198575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-certificates\") pod \"keda-admission-6c4d864b86-849qb\" (UID: \"87c88754-30df-4c85-94e0-1ce0f421e3c1\") " pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:30.201121 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:30.201095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/87c88754-30df-4c85-94e0-1ce0f421e3c1-certificates\") pod \"keda-admission-6c4d864b86-849qb\" (UID: \"87c88754-30df-4c85-94e0-1ce0f421e3c1\") " pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:30.319290 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:30.319238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:30.595376 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:30.595296    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-849qb"]
Sep 17 15:45:30.598644 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:45:30.598619    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c88754_30df_4c85_94e0_1ce0f421e3c1.slice/crio-ef644ef9c9dd763d94b782c54d9ea6ad09d8bbaf9e36dac7a0bf06032f35a558 WatchSource:0}: Error finding container ef644ef9c9dd763d94b782c54d9ea6ad09d8bbaf9e36dac7a0bf06032f35a558: Status 404 returned error can't find the container with id ef644ef9c9dd763d94b782c54d9ea6ad09d8bbaf9e36dac7a0bf06032f35a558
Sep 17 15:45:30.600299 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:30.600281    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:45:31.367378 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:31.367297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-849qb" event={"ID":"87c88754-30df-4c85-94e0-1ce0f421e3c1","Type":"ContainerStarted","Data":"ef644ef9c9dd763d94b782c54d9ea6ad09d8bbaf9e36dac7a0bf06032f35a558"}
Sep 17 15:45:33.375370 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:33.375342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-849qb" event={"ID":"87c88754-30df-4c85-94e0-1ce0f421e3c1","Type":"ContainerStarted","Data":"e4ccaf89f4cd77891ee53ed935cfa41699fb0f6baf216b3d77965c97cb4b15ea"}
Sep 17 15:45:33.375763 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:33.375473    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:45:33.392121 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:33.392078    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-849qb" podStartSLOduration=3.080870131 podStartE2EDuration="5.392061855s" totalImagesPullingTime="2.311191724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:33.390915671 +0000 UTC m=+418.233278476" watchObservedRunningTime="2026-09-17 15:45:33.392061855 +0000 UTC m=+418.234424682"
Sep 17 15:45:54.381757 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:45:54.381721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-849qb"
Sep 17 15:46:20.785765 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:20.785716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:20.841685 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:20.841641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:46.536272 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.536239    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:46.644663 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.644635    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-cj7ql"]
Sep 17 15:46:46.647677 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.647661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:46.650220 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.650195    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dl92p\""
Sep 17 15:46:46.650319 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.650289    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 15:46:46.650729 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.650715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 15:46:46.650770 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.650750    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 15:46:46.656919 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.656901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-cj7ql"]
Sep 17 15:46:46.663537 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.663518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn9t5\" (UniqueName: \"kubernetes.io/projected/2842e953-41b7-4eb3-8e8b-877129e52bf6-kube-api-access-pn9t5\") pod \"seaweedfs-64568bcd49-cj7ql\" (UID: \"2842e953-41b7-4eb3-8e8b-877129e52bf6\") " pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:46.663646 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.663590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2842e953-41b7-4eb3-8e8b-877129e52bf6-data\") pod \"seaweedfs-64568bcd49-cj7ql\" (UID: \"2842e953-41b7-4eb3-8e8b-877129e52bf6\") " pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:46.736419 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.736393    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:46.763950 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.763927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2842e953-41b7-4eb3-8e8b-877129e52bf6-data\") pod \"seaweedfs-64568bcd49-cj7ql\" (UID: \"2842e953-41b7-4eb3-8e8b-877129e52bf6\") " pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:46.764085 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.763977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn9t5\" (UniqueName: \"kubernetes.io/projected/2842e953-41b7-4eb3-8e8b-877129e52bf6-kube-api-access-pn9t5\") pod \"seaweedfs-64568bcd49-cj7ql\" (UID: \"2842e953-41b7-4eb3-8e8b-877129e52bf6\") " pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:46.764371 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.764348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2842e953-41b7-4eb3-8e8b-877129e52bf6-data\") pod \"seaweedfs-64568bcd49-cj7ql\" (UID: \"2842e953-41b7-4eb3-8e8b-877129e52bf6\") " pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:46.772240 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.772214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn9t5\" (UniqueName: \"kubernetes.io/projected/2842e953-41b7-4eb3-8e8b-877129e52bf6-kube-api-access-pn9t5\") pod \"seaweedfs-64568bcd49-cj7ql\" (UID: \"2842e953-41b7-4eb3-8e8b-877129e52bf6\") " pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:46.837061 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.836975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:46.933731 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.933699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:46.956594 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:46.956574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:47.035395 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:47.035371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:47.177329 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:47.177269    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:47.227458 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:47.227381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-cj7ql"]
Sep 17 15:46:47.233995 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:46:47.233953    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2842e953_41b7_4eb3_8e8b_877129e52bf6.slice/crio-2d1e682151500f50b0312a9362fd2d02c8788d46bba20bce2010af10d0bf0e6c WatchSource:0}: Error finding container 2d1e682151500f50b0312a9362fd2d02c8788d46bba20bce2010af10d0bf0e6c: Status 404 returned error can't find the container with id 2d1e682151500f50b0312a9362fd2d02c8788d46bba20bce2010af10d0bf0e6c
Sep 17 15:46:47.234427 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:47.234379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:47.441179 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:47.441076    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:47.635377 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:47.635338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-cj7ql" event={"ID":"2842e953-41b7-4eb3-8e8b-877129e52bf6","Type":"ContainerStarted","Data":"2d1e682151500f50b0312a9362fd2d02c8788d46bba20bce2010af10d0bf0e6c"}
Sep 17 15:46:50.648975 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:50.648939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-cj7ql" event={"ID":"2842e953-41b7-4eb3-8e8b-877129e52bf6","Type":"ContainerStarted","Data":"6cbf567fccfa553761ad1b2e5f6cbd33f1d6d5b8da4f96dc3d79a08d43b26514"}
Sep 17 15:46:50.649327 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:50.649030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:50.664662 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:50.664618    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-cj7ql" podStartSLOduration=1.394367168 podStartE2EDuration="4.664606114s" totalImagesPullingTime="3.270238946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:46:50.663928915 +0000 UTC m=+495.506291719" watchObservedRunningTime="2026-09-17 15:46:50.664606114 +0000 UTC m=+495.506968920"
Sep 17 15:46:53.063203 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:53.063171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:53.161843 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:53.161791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:55.740001 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:55.739925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:55.842275 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:55.842241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:46:56.655167 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:56.655140    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-cj7ql"
Sep 17 15:46:56.964374 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:46:56.964299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:04.071146 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:47:04.071117    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:47:04.168701 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:47:04.168631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:04.429382 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.429345    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-nmgt6"]
Sep 17 15:48:04.432598 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.432576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:04.434942 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.434921    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 15:48:04.434942 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.434935    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-7w6c5\""
Sep 17 15:48:04.435087 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.434921    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 15:48:04.443412 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.443391    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nmgt6"]
Sep 17 15:48:04.525648 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.525614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d52f465-876d-4092-b2b4-31e5119b5fd7-metrics-certs\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:04.525847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.525669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbfbp\" (UniqueName: \"kubernetes.io/projected/7d52f465-876d-4092-b2b4-31e5119b5fd7-kube-api-access-hbfbp\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:04.525847 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.525707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d52f465-876d-4092-b2b4-31e5119b5fd7-cert\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:04.626704 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.626670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d52f465-876d-4092-b2b4-31e5119b5fd7-cert\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:04.626891 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.626709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d52f465-876d-4092-b2b4-31e5119b5fd7-metrics-certs\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:04.626891 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.626752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfbp\" (UniqueName: \"kubernetes.io/projected/7d52f465-876d-4092-b2b4-31e5119b5fd7-kube-api-access-hbfbp\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:04.626891 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:48:04.626822    2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 15:48:04.626891 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:48:04.626884    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d52f465-876d-4092-b2b4-31e5119b5fd7-cert podName:7d52f465-876d-4092-b2b4-31e5119b5fd7 nodeName:}" failed. No retries permitted until 2026-09-17 15:48:05.126866455 +0000 UTC m=+569.969229241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7d52f465-876d-4092-b2b4-31e5119b5fd7-cert") pod "odh-model-controller-7b966d7984-nmgt6" (UID: "7d52f465-876d-4092-b2b4-31e5119b5fd7") : secret "odh-model-controller-webhook-cert" not found
Sep 17 15:48:04.629285 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.629266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d52f465-876d-4092-b2b4-31e5119b5fd7-metrics-certs\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:04.635136 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.635117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbfbp\" (UniqueName: \"kubernetes.io/projected/7d52f465-876d-4092-b2b4-31e5119b5fd7-kube-api-access-hbfbp\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:04.708056 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.707986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:04.807466 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.807437    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:04.908288 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:04.908252    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:05.130819 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:05.130769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d52f465-876d-4092-b2b4-31e5119b5fd7-cert\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:05.133206 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:05.133178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d52f465-876d-4092-b2b4-31e5119b5fd7-cert\") pod \"odh-model-controller-7b966d7984-nmgt6\" (UID: \"7d52f465-876d-4092-b2b4-31e5119b5fd7\") " pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:05.343440 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:05.343400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:05.614993 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:05.614970    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nmgt6"]
Sep 17 15:48:05.617059 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:48:05.617021    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d52f465_876d_4092_b2b4_31e5119b5fd7.slice/crio-7bbae54fee92a1527d905dd982afa23ee05961b4ed093dd19ca2711db0e5caf1 WatchSource:0}: Error finding container 7bbae54fee92a1527d905dd982afa23ee05961b4ed093dd19ca2711db0e5caf1: Status 404 returned error can't find the container with id 7bbae54fee92a1527d905dd982afa23ee05961b4ed093dd19ca2711db0e5caf1
Sep 17 15:48:05.913752 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:05.913719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nmgt6" event={"ID":"7d52f465-876d-4092-b2b4-31e5119b5fd7","Type":"ContainerStarted","Data":"7bbae54fee92a1527d905dd982afa23ee05961b4ed093dd19ca2711db0e5caf1"}
Sep 17 15:48:05.948056 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:05.948028    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:06.045640 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:06.045614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:48:08.932027 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:08.931993    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nmgt6" event={"ID":"7d52f465-876d-4092-b2b4-31e5119b5fd7","Type":"ContainerStarted","Data":"c4ab40989b486d2e1a4233041d0d267e56122f744b807f1bfb2d24a296226f6f"}
Sep 17 15:48:08.932514 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:08.932094    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:08.954745 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:08.954696    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-nmgt6" podStartSLOduration=2.122003367 podStartE2EDuration="4.954681937s" totalImagesPullingTime="2.83267857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:08.953383566 +0000 UTC m=+573.795746373" watchObservedRunningTime="2026-09-17 15:48:08.954681937 +0000 UTC m=+573.797044754"
Sep 17 15:48:19.937726 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:19.937696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-nmgt6"
Sep 17 15:48:35.603378 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:35.603345    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:48:35.604334 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:48:35.604303    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:49:41.031745 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.031707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:41.034479 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.034456    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"]
Sep 17 15:49:41.037496 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.037482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:41.039538 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.039517    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 15:49:41.045778 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.045756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"]
Sep 17 15:49:41.129025 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.128989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j578h\" (UniqueName: \"kubernetes.io/projected/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-kube-api-access-j578h\") pod \"seaweedfs-tls-custom-6644bf5d6c-jbcm9\" (UID: \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:41.129178 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.129057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-jbcm9\" (UID: \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:41.229657 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.229622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j578h\" (UniqueName: \"kubernetes.io/projected/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-kube-api-access-j578h\") pod \"seaweedfs-tls-custom-6644bf5d6c-jbcm9\" (UID: \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:41.229850 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.229689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-jbcm9\" (UID: \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:41.230047 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.230031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-jbcm9\" (UID: \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:41.238608 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.238583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j578h\" (UniqueName: \"kubernetes.io/projected/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-kube-api-access-j578h\") pod \"seaweedfs-tls-custom-6644bf5d6c-jbcm9\" (UID: \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:41.346941 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.346865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:41.617735 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:41.617704    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"]
Sep 17 15:49:41.621326 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:49:41.621298    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf296752c_5d0a_400c_b41b_a9e0f82ed0d1.slice/crio-cd3af24dcaf5c393cf23124531aa69181be33d1948c374909435a459d0c54796 WatchSource:0}: Error finding container cd3af24dcaf5c393cf23124531aa69181be33d1948c374909435a459d0c54796: Status 404 returned error can't find the container with id cd3af24dcaf5c393cf23124531aa69181be33d1948c374909435a459d0c54796
Sep 17 15:49:42.270105 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:42.270020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9" event={"ID":"f296752c-5d0a-400c-b41b-a9e0f82ed0d1","Type":"ContainerStarted","Data":"15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4"}
Sep 17 15:49:42.270105 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:42.270076    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9" event={"ID":"f296752c-5d0a-400c-b41b-a9e0f82ed0d1","Type":"ContainerStarted","Data":"cd3af24dcaf5c393cf23124531aa69181be33d1948c374909435a459d0c54796"}
Sep 17 15:49:42.284548 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:42.284497    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9" podStartSLOduration=1.001020191 podStartE2EDuration="1.284476172s" totalImagesPullingTime="283.455981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:42.283981526 +0000 UTC m=+667.126344332" watchObservedRunningTime="2026-09-17 15:49:42.284476172 +0000 UTC m=+667.126838976"
Sep 17 15:49:43.271999 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:43.271964    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"]
Sep 17 15:49:44.277834 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:44.277765    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9" podUID="f296752c-5d0a-400c-b41b-a9e0f82ed0d1" containerName="seaweedfs-tls-custom" containerID="cri-o://15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4" gracePeriod=30
Sep 17 15:49:45.510785 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:45.510761    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:45.663817 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:45.663762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-data\") pod \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\" (UID: \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\") "
Sep 17 15:49:45.663995 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:45.663859    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j578h\" (UniqueName: \"kubernetes.io/projected/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-kube-api-access-j578h\") pod \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\" (UID: \"f296752c-5d0a-400c-b41b-a9e0f82ed0d1\") "
Sep 17 15:49:45.665048 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:45.665020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-data" (OuterVolumeSpecName: "data") pod "f296752c-5d0a-400c-b41b-a9e0f82ed0d1" (UID: "f296752c-5d0a-400c-b41b-a9e0f82ed0d1"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:45.666112 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:45.666082    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-kube-api-access-j578h" (OuterVolumeSpecName: "kube-api-access-j578h") pod "f296752c-5d0a-400c-b41b-a9e0f82ed0d1" (UID: "f296752c-5d0a-400c-b41b-a9e0f82ed0d1"). InnerVolumeSpecName "kube-api-access-j578h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:45.765356 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:45.765314    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j578h\" (UniqueName: \"kubernetes.io/projected/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-kube-api-access-j578h\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:49:45.765356 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:45.765347    2580 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f296752c-5d0a-400c-b41b-a9e0f82ed0d1-data\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:49:46.286011 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.285961    2580 generic.go:356] "Generic (PLEG): container finished" podID="f296752c-5d0a-400c-b41b-a9e0f82ed0d1" containerID="15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4" exitCode=0
Sep 17 15:49:46.286208 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.286044    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"
Sep 17 15:49:46.286208 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.286045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9" event={"ID":"f296752c-5d0a-400c-b41b-a9e0f82ed0d1","Type":"ContainerDied","Data":"15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4"}
Sep 17 15:49:46.286208 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.286192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9" event={"ID":"f296752c-5d0a-400c-b41b-a9e0f82ed0d1","Type":"ContainerDied","Data":"cd3af24dcaf5c393cf23124531aa69181be33d1948c374909435a459d0c54796"}
Sep 17 15:49:46.286393 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.286228    2580 scope.go:117] "RemoveContainer" containerID="15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4"
Sep 17 15:49:46.295084 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.295070    2580 scope.go:117] "RemoveContainer" containerID="15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4"
Sep 17 15:49:46.295348 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:49:46.295328    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4\": container with ID starting with 15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4 not found: ID does not exist" containerID="15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4"
Sep 17 15:49:46.295429 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.295353    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4"} err="failed to get container status \"15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4\": rpc error: code = NotFound desc = could not find container \"15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4\": container with ID starting with 15bb08c921925508c96e4d8bb9cd85496e83fb1331eee4f2dec7da05a40fe7c4 not found: ID does not exist"
Sep 17 15:49:46.302078 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.302057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"]
Sep 17 15:49:46.305708 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.305687    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-jbcm9"]
Sep 17 15:49:46.344718 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.344694    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"]
Sep 17 15:49:46.344982 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.344970    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f296752c-5d0a-400c-b41b-a9e0f82ed0d1" containerName="seaweedfs-tls-custom"
Sep 17 15:49:46.345033 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.344985    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f296752c-5d0a-400c-b41b-a9e0f82ed0d1" containerName="seaweedfs-tls-custom"
Sep 17 15:49:46.345063 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.345049    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f296752c-5d0a-400c-b41b-a9e0f82ed0d1" containerName="seaweedfs-tls-custom"
Sep 17 15:49:46.349081 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.349067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.350958 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.350938    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 15:49:46.351059 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.351015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 17 15:49:46.354343 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.354315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"]
Sep 17 15:49:46.470114 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.470081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xvc\" (UniqueName: \"kubernetes.io/projected/c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1-kube-api-access-p2xvc\") pod \"seaweedfs-tls-custom-8b44d767d-sw9j5\" (UID: \"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.470260 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.470137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-sw9j5\" (UID: \"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.470260 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.470183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1-data\") pod \"seaweedfs-tls-custom-8b44d767d-sw9j5\" (UID: \"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.570761 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.570678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xvc\" (UniqueName: \"kubernetes.io/projected/c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1-kube-api-access-p2xvc\") pod \"seaweedfs-tls-custom-8b44d767d-sw9j5\" (UID: \"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.570761 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.570740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-sw9j5\" (UID: \"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.571156 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.570766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1-data\") pod \"seaweedfs-tls-custom-8b44d767d-sw9j5\" (UID: \"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.571156 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.571122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1-data\") pod \"seaweedfs-tls-custom-8b44d767d-sw9j5\" (UID: \"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.573362 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.573342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-sw9j5\" (UID: \"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.579728 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.579705    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xvc\" (UniqueName: \"kubernetes.io/projected/c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1-kube-api-access-p2xvc\") pod \"seaweedfs-tls-custom-8b44d767d-sw9j5\" (UID: \"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.658566 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.658534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"
Sep 17 15:49:46.925017 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:46.924988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-sw9j5"]
Sep 17 15:49:46.927910 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:49:46.927877    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f2c7a9_3e90_4088_81ab_d15bcbbf76b1.slice/crio-d5931bbbaf82143cf45b226f0a2382a6165fa8ab45c21aa84f0aae8356bf108f WatchSource:0}: Error finding container d5931bbbaf82143cf45b226f0a2382a6165fa8ab45c21aa84f0aae8356bf108f: Status 404 returned error can't find the container with id d5931bbbaf82143cf45b226f0a2382a6165fa8ab45c21aa84f0aae8356bf108f
Sep 17 15:49:47.292773 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:47.292741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5" event={"ID":"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1","Type":"ContainerStarted","Data":"bd6cecd082468303935a7eeb366ac3a3d6efd7d59601281f5de5cd58a6568a81"}
Sep 17 15:49:47.292998 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:47.292785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5" event={"ID":"c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1","Type":"ContainerStarted","Data":"d5931bbbaf82143cf45b226f0a2382a6165fa8ab45c21aa84f0aae8356bf108f"}
Sep 17 15:49:47.308821 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:47.308765    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-sw9j5" podStartSLOduration=1.05225285 podStartE2EDuration="1.308746088s" totalImagesPullingTime="256.493238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:47.307924316 +0000 UTC m=+672.150287122" watchObservedRunningTime="2026-09-17 15:49:47.308746088 +0000 UTC m=+672.151108894"
Sep 17 15:49:47.674580 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:47.674546    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f296752c-5d0a-400c-b41b-a9e0f82ed0d1" path="/var/lib/kubelet/pods/f296752c-5d0a-400c-b41b-a9e0f82ed0d1/volumes"
Sep 17 15:49:58.893888 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:58.893853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:49:58.900256 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:58.900234    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"]
Sep 17 15:49:58.904153 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:58.904139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:58.906340 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:58.906313    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 15:49:58.906441 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:58.906311    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 17 15:49:58.911698 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:58.911200    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"]
Sep 17 15:49:59.073627 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.073593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/36caaf15-691b-42cb-b30b-5962cb72e9ec-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-wr9rh\" (UID: \"36caaf15-691b-42cb-b30b-5962cb72e9ec\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.073791 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.073656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36caaf15-691b-42cb-b30b-5962cb72e9ec-data\") pod \"seaweedfs-tls-serving-67b7477c6d-wr9rh\" (UID: \"36caaf15-691b-42cb-b30b-5962cb72e9ec\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.073791 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.073677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthg5\" (UniqueName: \"kubernetes.io/projected/36caaf15-691b-42cb-b30b-5962cb72e9ec-kube-api-access-jthg5\") pod \"seaweedfs-tls-serving-67b7477c6d-wr9rh\" (UID: \"36caaf15-691b-42cb-b30b-5962cb72e9ec\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.174439 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.174353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/36caaf15-691b-42cb-b30b-5962cb72e9ec-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-wr9rh\" (UID: \"36caaf15-691b-42cb-b30b-5962cb72e9ec\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.174439 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.174417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36caaf15-691b-42cb-b30b-5962cb72e9ec-data\") pod \"seaweedfs-tls-serving-67b7477c6d-wr9rh\" (UID: \"36caaf15-691b-42cb-b30b-5962cb72e9ec\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.174439 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.174438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthg5\" (UniqueName: \"kubernetes.io/projected/36caaf15-691b-42cb-b30b-5962cb72e9ec-kube-api-access-jthg5\") pod \"seaweedfs-tls-serving-67b7477c6d-wr9rh\" (UID: \"36caaf15-691b-42cb-b30b-5962cb72e9ec\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.174880 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.174859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36caaf15-691b-42cb-b30b-5962cb72e9ec-data\") pod \"seaweedfs-tls-serving-67b7477c6d-wr9rh\" (UID: \"36caaf15-691b-42cb-b30b-5962cb72e9ec\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.176987 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.176954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/36caaf15-691b-42cb-b30b-5962cb72e9ec-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-wr9rh\" (UID: \"36caaf15-691b-42cb-b30b-5962cb72e9ec\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.183420 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.183389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthg5\" (UniqueName: \"kubernetes.io/projected/36caaf15-691b-42cb-b30b-5962cb72e9ec-kube-api-access-jthg5\") pod \"seaweedfs-tls-serving-67b7477c6d-wr9rh\" (UID: \"36caaf15-691b-42cb-b30b-5962cb72e9ec\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.215056 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.215029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"
Sep 17 15:49:59.484110 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:49:59.484087    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh"]
Sep 17 15:49:59.486479 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:49:59.486455    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36caaf15_691b_42cb_b30b_5962cb72e9ec.slice/crio-57060d1e128c626ba0cc433f51d3e56bedbe34aec9b2cfe5c566c243541633a2 WatchSource:0}: Error finding container 57060d1e128c626ba0cc433f51d3e56bedbe34aec9b2cfe5c566c243541633a2: Status 404 returned error can't find the container with id 57060d1e128c626ba0cc433f51d3e56bedbe34aec9b2cfe5c566c243541633a2
Sep 17 15:50:00.344277 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:00.344242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh" event={"ID":"36caaf15-691b-42cb-b30b-5962cb72e9ec","Type":"ContainerStarted","Data":"2297026b32c7182a76bc1e95d80edeccd0db3a51c73323ce6c85054455fe1ff9"}
Sep 17 15:50:00.344644 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:00.344295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh" event={"ID":"36caaf15-691b-42cb-b30b-5962cb72e9ec","Type":"ContainerStarted","Data":"57060d1e128c626ba0cc433f51d3e56bedbe34aec9b2cfe5c566c243541633a2"}
Sep 17 15:50:00.359663 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:00.359615    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-wr9rh" podStartSLOduration=2.024157942 podStartE2EDuration="2.359598328s" totalImagesPullingTime="335.440386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:00.358614084 +0000 UTC m=+685.200976888" watchObservedRunningTime="2026-09-17 15:50:00.359598328 +0000 UTC m=+685.201961134"
Sep 17 15:50:00.670600 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:00.670569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:14.146742 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:14.146705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:14.247681 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:14.247649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:19.501657 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:19.501602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:19.571323 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:19.571287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:25.694893 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:25.694844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:25.909114 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:25.909079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:26.110520 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:26.110483    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:26.297705 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:26.297669    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:35.584142 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:35.584059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:40.781902 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:40.781864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:41.032113 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:41.031996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:50:55.302327 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:50:55.302285    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:51:32.344626 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:51:32.344588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:51:32.409821 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:51:32.409755    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:51:37.571583 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:51:37.571554    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:20.860487 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:20.860424    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:20.978291 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:20.978253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:21.201758 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:21.201665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:41.449107 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:41.449058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:46.528727 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.528698    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:46.567748 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.567723    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"]
Sep 17 15:52:46.570031 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.570016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.572525 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.572498    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 15:52:46.572819 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.572673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 15:52:46.572917 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.572899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 15:52:46.573107 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.572754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 15:52:46.573229 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.573210    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:52:46.573412 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.573395    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 15:52:46.573527 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.573509    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 15:52:46.583477 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.583454    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"]
Sep 17 15:52:46.645295 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.645257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.645295 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.645290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.645499 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.645331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.645499 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.645370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.645499 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.645449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grjp2\" (UniqueName: \"kubernetes.io/projected/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-api-access-grjp2\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.690913 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.690872    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:52:46.746032 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.746000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.746159 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.746053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.746159 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.746091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grjp2\" (UniqueName: \"kubernetes.io/projected/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-api-access-grjp2\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.746159 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.746139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.746159 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.746155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.746610 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.746583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.746786 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.746767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.746786 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.746776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.748697 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.748681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.756019 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.755999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grjp2\" (UniqueName: \"kubernetes.io/projected/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-api-access-grjp2\") pod \"isvc-predictive-xgboost-predictor-695f978978-qwkms\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:46.882983 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:46.882945    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:52:47.158763 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:47.158737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"]
Sep 17 15:52:47.161140 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:52:47.161104    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a6cafa_d940_45c3_b4f7_a32cb5da60fe.slice/crio-fbcd0e0459a991656135c586160f300e2d04ede37eb31d256ca90a5c08ecf149 WatchSource:0}: Error finding container fbcd0e0459a991656135c586160f300e2d04ede37eb31d256ca90a5c08ecf149: Status 404 returned error can't find the container with id fbcd0e0459a991656135c586160f300e2d04ede37eb31d256ca90a5c08ecf149
Sep 17 15:52:47.163132 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:47.163114    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:52:47.970423 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:47.970283    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" event={"ID":"98a6cafa-d940-45c3-b4f7-a32cb5da60fe","Type":"ContainerStarted","Data":"fbcd0e0459a991656135c586160f300e2d04ede37eb31d256ca90a5c08ecf149"}
Sep 17 15:52:50.986164 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:50.986085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" event={"ID":"98a6cafa-d940-45c3-b4f7-a32cb5da60fe","Type":"ContainerStarted","Data":"be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51"}
Sep 17 15:52:55.003123 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:55.003087    2580 generic.go:356] "Generic (PLEG): container finished" podID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerID="be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51" exitCode=0
Sep 17 15:52:55.003501 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:52:55.003147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" event={"ID":"98a6cafa-d940-45c3-b4f7-a32cb5da60fe","Type":"ContainerDied","Data":"be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51"}
Sep 17 15:53:18.119390 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:18.119355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" event={"ID":"98a6cafa-d940-45c3-b4f7-a32cb5da60fe","Type":"ContainerStarted","Data":"c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3"}
Sep 17 15:53:21.134443 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:21.134408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" event={"ID":"98a6cafa-d940-45c3-b4f7-a32cb5da60fe","Type":"ContainerStarted","Data":"cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc"}
Sep 17 15:53:21.134839 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:21.134573    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:53:21.162027 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:21.161975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podStartSLOduration=2.308832788 podStartE2EDuration="35.161959706s" totalImagesPullingTime="29.003456093s" totalInitContainerRuntime="3.849670825s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:52:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:21.160480543 +0000 UTC m=+886.002843349" watchObservedRunningTime="2026-09-17 15:53:21.161959706 +0000 UTC m=+886.004322570"
Sep 17 15:53:22.138221 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:22.138175    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:53:22.139688 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:22.139658    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:53:23.142566 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:23.142524    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:53:28.147577 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:28.147499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:53:28.148084 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:28.148052    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:53:35.630873 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:35.630844    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:53:35.631566 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:35.631097    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:53:38.148213 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:38.148162    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:53:46.170779 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:46.170742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:46.305886 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:46.305844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:46.555225 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:46.555140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:53:48.148596 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:48.148556    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:53:58.148040 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:53:58.147993    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:06.220042 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:06.220005    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:06.347505 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:06.347463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:08.148622 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:08.148573    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:18.148642 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:18.148594    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:28.148303 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:28.148255    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:38.148942 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:38.148913    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:54:47.112903 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:47.112867    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"]
Sep 17 15:54:47.113292 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:47.113128    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" containerID="cri-o://c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3" gracePeriod=30
Sep 17 15:54:47.113292 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:47.113187    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kube-rbac-proxy" containerID="cri-o://cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc" gracePeriod=30
Sep 17 15:54:47.152135 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:47.152105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:47.474643 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:47.474566    2580 generic.go:356] "Generic (PLEG): container finished" podID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerID="cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc" exitCode=2
Sep 17 15:54:47.474643 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:47.474618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" event={"ID":"98a6cafa-d940-45c3-b4f7-a32cb5da60fe","Type":"ContainerDied","Data":"cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc"}
Sep 17 15:54:48.143576 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:48.143539    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 15:54:48.148944 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:48.148921    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:51.857937 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.857915    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:54:51.866096 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.866077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kserve-provision-location\") pod \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") "
Sep 17 15:54:51.866222 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.866112    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-cabundle-cert\") pod \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") "
Sep 17 15:54:51.866222 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.866131    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-proxy-tls\") pod \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") "
Sep 17 15:54:51.866222 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.866154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grjp2\" (UniqueName: \"kubernetes.io/projected/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-api-access-grjp2\") pod \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") "
Sep 17 15:54:51.866387 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.866293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-rbac-proxy-sar-config\") pod \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\" (UID: \"98a6cafa-d940-45c3-b4f7-a32cb5da60fe\") "
Sep 17 15:54:51.866441 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.866410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98a6cafa-d940-45c3-b4f7-a32cb5da60fe" (UID: "98a6cafa-d940-45c3-b4f7-a32cb5da60fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:51.866541 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.866525    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.866623 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.866547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98a6cafa-d940-45c3-b4f7-a32cb5da60fe" (UID: "98a6cafa-d940-45c3-b4f7-a32cb5da60fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:51.866659 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.866634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98a6cafa-d940-45c3-b4f7-a32cb5da60fe" (UID: "98a6cafa-d940-45c3-b4f7-a32cb5da60fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:51.868410 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.868385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-api-access-grjp2" (OuterVolumeSpecName: "kube-api-access-grjp2") pod "98a6cafa-d940-45c3-b4f7-a32cb5da60fe" (UID: "98a6cafa-d940-45c3-b4f7-a32cb5da60fe"). InnerVolumeSpecName "kube-api-access-grjp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:51.868410 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.868398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98a6cafa-d940-45c3-b4f7-a32cb5da60fe" (UID: "98a6cafa-d940-45c3-b4f7-a32cb5da60fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:51.966826 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.966751    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.966826 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.966773    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.966826 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.966782    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grjp2\" (UniqueName: \"kubernetes.io/projected/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-api-access-grjp2\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.966826 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:51.966790    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a6cafa-d940-45c3-b4f7-a32cb5da60fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:54:52.275239 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.275165    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:52.309224 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.309183    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"]
Sep 17 15:54:52.309478 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.309467    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kube-rbac-proxy"
Sep 17 15:54:52.309527 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.309481    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kube-rbac-proxy"
Sep 17 15:54:52.309527 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.309490    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="storage-initializer"
Sep 17 15:54:52.309527 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.309496    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="storage-initializer"
Sep 17 15:54:52.309527 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.309505    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container"
Sep 17 15:54:52.309527 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.309511    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container"
Sep 17 15:54:52.309681 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.309561    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kube-rbac-proxy"
Sep 17 15:54:52.309681 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.309571    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerName="kserve-container"
Sep 17 15:54:52.311672 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.311656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.316311 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.316287    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 15:54:52.316412 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.316326    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 15:54:52.325700 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.325678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"]
Sep 17 15:54:52.369615 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.369589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.369711 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.369629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.369711 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.369647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.369795 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.369738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.369795 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.369762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6grk\" (UniqueName: \"kubernetes.io/projected/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-api-access-c6grk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.470921 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.470889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.470921 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.470920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6grk\" (UniqueName: \"kubernetes.io/projected/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-api-access-c6grk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.471118 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.470942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.471118 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.470984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.471118 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.471009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.471332 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.471313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.471646 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.471628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.471694 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.471649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.473539 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.473520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.479520 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.479496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6grk\" (UniqueName: \"kubernetes.io/projected/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-api-access-c6grk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.494408 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.494384    2580 generic.go:356] "Generic (PLEG): container finished" podID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" containerID="c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3" exitCode=0
Sep 17 15:54:52.494512 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.494451    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"
Sep 17 15:54:52.494512 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.494466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" event={"ID":"98a6cafa-d940-45c3-b4f7-a32cb5da60fe","Type":"ContainerDied","Data":"c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3"}
Sep 17 15:54:52.494636 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.494526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms" event={"ID":"98a6cafa-d940-45c3-b4f7-a32cb5da60fe","Type":"ContainerDied","Data":"fbcd0e0459a991656135c586160f300e2d04ede37eb31d256ca90a5c08ecf149"}
Sep 17 15:54:52.494636 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.494552    2580 scope.go:117] "RemoveContainer" containerID="cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc"
Sep 17 15:54:52.505760 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.505742    2580 scope.go:117] "RemoveContainer" containerID="c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3"
Sep 17 15:54:52.513319 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.513302    2580 scope.go:117] "RemoveContainer" containerID="be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51"
Sep 17 15:54:52.517088 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.517069    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"]
Sep 17 15:54:52.521156 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.521127    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:52.521232 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.521160    2580 scope.go:117] "RemoveContainer" containerID="cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc"
Sep 17 15:54:52.521457 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:54:52.521439    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc\": container with ID starting with cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc not found: ID does not exist" containerID="cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc"
Sep 17 15:54:52.521523 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.521462    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc"} err="failed to get container status \"cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc\": rpc error: code = NotFound desc = could not find container \"cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc\": container with ID starting with cd2f33e216d21a624c68a943aca9666ca736f49c42fa8562f34aaa3c94a2bfbc not found: ID does not exist"
Sep 17 15:54:52.521523 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.521478    2580 scope.go:117] "RemoveContainer" containerID="c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3"
Sep 17 15:54:52.521741 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:54:52.521722    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3\": container with ID starting with c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3 not found: ID does not exist" containerID="c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3"
Sep 17 15:54:52.521820 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.521746    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3"} err="failed to get container status \"c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3\": rpc error: code = NotFound desc = could not find container \"c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3\": container with ID starting with c2648fb152357dd8213cbc624f996ed779e13a2a1c062d29fa6b19a3aa085ed3 not found: ID does not exist"
Sep 17 15:54:52.521820 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.521761    2580 scope.go:117] "RemoveContainer" containerID="be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51"
Sep 17 15:54:52.522006 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.521988    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-qwkms"]
Sep 17 15:54:52.522050 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:54:52.522019    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51\": container with ID starting with be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51 not found: ID does not exist" containerID="be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51"
Sep 17 15:54:52.522050 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.522040    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51"} err="failed to get container status \"be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51\": rpc error: code = NotFound desc = could not find container \"be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51\": container with ID starting with be42d8627744d52bdda210cdec96d25ae0bcd6d19167a6d40cb1b578f40dbc51 not found: ID does not exist"
Sep 17 15:54:52.621958 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.621932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:52.896859 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:52.896672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"]
Sep 17 15:54:52.899285 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:54:52.899257    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1bf1e8_7332_47c2_b1cb_09bebe39c62a.slice/crio-189ac3f2ac37aab1ba875333bcfc320fe05f4c2644856654a555217034344869 WatchSource:0}: Error finding container 189ac3f2ac37aab1ba875333bcfc320fe05f4c2644856654a555217034344869: Status 404 returned error can't find the container with id 189ac3f2ac37aab1ba875333bcfc320fe05f4c2644856654a555217034344869
Sep 17 15:54:53.501290 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:53.501255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" event={"ID":"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a","Type":"ContainerStarted","Data":"202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35"}
Sep 17 15:54:53.501446 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:53.501311    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" event={"ID":"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a","Type":"ContainerStarted","Data":"189ac3f2ac37aab1ba875333bcfc320fe05f4c2644856654a555217034344869"}
Sep 17 15:54:53.674447 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:53.674416    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a6cafa-d940-45c3-b4f7-a32cb5da60fe" path="/var/lib/kubelet/pods/98a6cafa-d940-45c3-b4f7-a32cb5da60fe/volumes"
Sep 17 15:54:57.516860 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:57.516818    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerID="202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35" exitCode=0
Sep 17 15:54:57.517283 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:57.516892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" event={"ID":"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a","Type":"ContainerDied","Data":"202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35"}
Sep 17 15:54:58.522496 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:58.522468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" event={"ID":"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a","Type":"ContainerStarted","Data":"9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74"}
Sep 17 15:54:58.522856 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:58.522527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" event={"ID":"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a","Type":"ContainerStarted","Data":"c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54"}
Sep 17 15:54:58.522856 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:58.522732    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:58.544190 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:58.544138    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podStartSLOduration=2.466530335 podStartE2EDuration="6.544119515s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.07758918s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:58.543680138 +0000 UTC m=+983.386042945" watchObservedRunningTime="2026-09-17 15:54:58.544119515 +0000 UTC m=+983.386482323"
Sep 17 15:54:59.526508 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:59.526468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:54:59.527672 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:54:59.527647    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:55:00.530586 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:00.530547    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:55:05.535463 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:05.535435    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:55:05.535993 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:05.535964    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:55:15.536198 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:15.536157    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:55:25.536360 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:25.536317    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:55:35.536485 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:35.536447    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:55:41.403067 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.403021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:41.531878 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.531849    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"]
Sep 17 15:55:41.538866 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.538844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.541208 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.541179    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 15:55:41.541208 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.541203    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 15:55:41.547868 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.547846    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"]
Sep 17 15:55:41.648375 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.648350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.648521 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.648380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.648521 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.648399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.648521 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.648422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.648521 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.648498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkw9c\" (UniqueName: \"kubernetes.io/projected/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-api-access-nkw9c\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.749382 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.749314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.749382 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.749350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.749382 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.749380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkw9c\" (UniqueName: \"kubernetes.io/projected/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-api-access-nkw9c\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.749619 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.749416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.749619 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.749434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.749866 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.749842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.750038 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.750010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.750167 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.750148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.752002 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.751985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.759850 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.759830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkw9c\" (UniqueName: \"kubernetes.io/projected/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-api-access-nkw9c\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7j94v\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:41.849184 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:41.849153    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:55:42.121308 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:42.121281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"]
Sep 17 15:55:42.123791 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:55:42.123747    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea181dc_f8c3_4ed0_b6e8_db485d5196d7.slice/crio-4804f9c121f16219e4d7c594d69c1b4b5166966ab669e4f50ce3a422178ce111 WatchSource:0}: Error finding container 4804f9c121f16219e4d7c594d69c1b4b5166966ab669e4f50ce3a422178ce111: Status 404 returned error can't find the container with id 4804f9c121f16219e4d7c594d69c1b4b5166966ab669e4f50ce3a422178ce111
Sep 17 15:55:42.696298 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:42.696265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" event={"ID":"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7","Type":"ContainerStarted","Data":"e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e"}
Sep 17 15:55:42.696688 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:42.696323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" event={"ID":"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7","Type":"ContainerStarted","Data":"4804f9c121f16219e4d7c594d69c1b4b5166966ab669e4f50ce3a422178ce111"}
Sep 17 15:55:42.956086 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:42.956001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:45.536677 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:45.536637    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:55:49.724094 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:49.724019    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerID="e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e" exitCode=0
Sep 17 15:55:49.724094 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:49.724058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" event={"ID":"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7","Type":"ContainerDied","Data":"e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e"}
Sep 17 15:55:55.535955 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:55:55.535909    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:56:05.535952 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:05.535915    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:56:15.536249 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:15.536206    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:56:21.676508 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:21.676480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:56:23.003627 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:23.003579    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"]
Sep 17 15:56:23.004027 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:23.003860    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container" containerID="cri-o://c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54" gracePeriod=30
Sep 17 15:56:23.004027 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:23.003917    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kube-rbac-proxy" containerID="cri-o://9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74" gracePeriod=30
Sep 17 15:56:23.027585 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:23.027556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:23.875448 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:23.875416    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerID="9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74" exitCode=2
Sep 17 15:56:23.875633 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:23.875494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" event={"ID":"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a","Type":"ContainerDied","Data":"9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74"}
Sep 17 15:56:25.531078 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:25.531035    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 15:56:28.215563 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:28.215460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:28.423990 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:28.423941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:29.567221 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.567194    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:56:29.652424 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.652387    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-proxy-tls\") pod \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") "
Sep 17 15:56:29.652424 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.652429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6grk\" (UniqueName: \"kubernetes.io/projected/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-api-access-c6grk\") pod \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") "
Sep 17 15:56:29.652676 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.652463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-rbac-proxy-sar-config\") pod \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") "
Sep 17 15:56:29.652676 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.652492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kserve-provision-location\") pod \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") "
Sep 17 15:56:29.652676 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.652578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-cabundle-cert\") pod \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\" (UID: \"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a\") "
Sep 17 15:56:29.652934 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.652900    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" (UID: "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:29.652934 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.652912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" (UID: "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:29.653096 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.652975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" (UID: "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:29.654693 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.654675    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" (UID: "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:29.654780 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.654731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-api-access-c6grk" (OuterVolumeSpecName: "kube-api-access-c6grk") pod "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" (UID: "7b1bf1e8-7332-47c2-b1cb-09bebe39c62a"). InnerVolumeSpecName "kube-api-access-c6grk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:29.754091 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.754054    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.754091 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.754087    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6grk\" (UniqueName: \"kubernetes.io/projected/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-api-access-c6grk\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.754091 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.754096    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.754091 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.754105    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.754440 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.754117    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.906749 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.906714    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerID="c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54" exitCode=0
Sep 17 15:56:29.906918 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.906775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" event={"ID":"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a","Type":"ContainerDied","Data":"c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54"}
Sep 17 15:56:29.906918 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.906843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92" event={"ID":"7b1bf1e8-7332-47c2-b1cb-09bebe39c62a","Type":"ContainerDied","Data":"189ac3f2ac37aab1ba875333bcfc320fe05f4c2644856654a555217034344869"}
Sep 17 15:56:29.906918 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.906868    2580 scope.go:117] "RemoveContainer" containerID="9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74"
Sep 17 15:56:29.906918 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.906777    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"
Sep 17 15:56:29.916369 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.916351    2580 scope.go:117] "RemoveContainer" containerID="c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54"
Sep 17 15:56:29.926153 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.926131    2580 scope.go:117] "RemoveContainer" containerID="202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35"
Sep 17 15:56:29.933423 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.933402    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"]
Sep 17 15:56:29.936112 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.936001    2580 scope.go:117] "RemoveContainer" containerID="9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74"
Sep 17 15:56:29.936740 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:56:29.936706    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74\": container with ID starting with 9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74 not found: ID does not exist" containerID="9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74"
Sep 17 15:56:29.936892 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.936742    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74"} err="failed to get container status \"9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74\": rpc error: code = NotFound desc = could not find container \"9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74\": container with ID starting with 9b42a67a38a7785aa8bd68458191b5fbe9c23bdb672b2825271378f20ed3cf74 not found: ID does not exist"
Sep 17 15:56:29.936892 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.936768    2580 scope.go:117] "RemoveContainer" containerID="c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54"
Sep 17 15:56:29.937114 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:56:29.937092    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54\": container with ID starting with c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54 not found: ID does not exist" containerID="c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54"
Sep 17 15:56:29.937181 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.937123    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54"} err="failed to get container status \"c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54\": rpc error: code = NotFound desc = could not find container \"c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54\": container with ID starting with c4f91963fb2ea37b387694eeaeee0d50272b6b0a76674c9d10ed3b686a5a3c54 not found: ID does not exist"
Sep 17 15:56:29.937181 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.937143    2580 scope.go:117] "RemoveContainer" containerID="202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35"
Sep 17 15:56:29.937433 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:56:29.937407    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35\": container with ID starting with 202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35 not found: ID does not exist" containerID="202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35"
Sep 17 15:56:29.937523 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.937435    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35"} err="failed to get container status \"202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35\": rpc error: code = NotFound desc = could not find container \"202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35\": container with ID starting with 202b9109d06d8d3621cd3596cba283566d7e7e53bb939a08251072000ea4ae35 not found: ID does not exist"
Sep 17 15:56:29.937523 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:29.937444    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-ggv92"]
Sep 17 15:56:31.674879 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:56:31.674845    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" path="/var/lib/kubelet/pods/7b1bf1e8-7332-47c2-b1cb-09bebe39c62a/volumes"
Sep 17 15:57:56.281410 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:57:56.281378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" event={"ID":"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7","Type":"ContainerStarted","Data":"8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b"}
Sep 17 15:57:56.281953 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:57:56.281438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" event={"ID":"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7","Type":"ContainerStarted","Data":"eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f"}
Sep 17 15:57:56.281953 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:57:56.281508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:57:56.301412 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:57:56.301370    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" podStartSLOduration=2.327045474 podStartE2EDuration="2m15.301355496s" totalImagesPullingTime="2m5.698651886s" totalInitContainerRuntime="7.275658136s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:56.299059269 +0000 UTC m=+1161.141422071" watchObservedRunningTime="2026-09-17 15:57:56.301355496 +0000 UTC m=+1161.143718298"
Sep 17 15:57:57.287198 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:57:57.287122    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:57:57.288248 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:57:57.288214    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:57:58.292409 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:57:58.292353    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:57:58.382885 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:57:58.382849    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:03.297789 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:03.297761    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:58:03.298504 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:03.298479    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:58:03.563755 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:03.563643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:03.900893 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:03.900848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:13.298987 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:13.298953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:58:27.820042 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:27.820001    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"]
Sep 17 15:58:27.820566 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:27.820234    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kserve-container" containerID="cri-o://eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f" gracePeriod=30
Sep 17 15:58:27.820566 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:27.820288    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kube-rbac-proxy" containerID="cri-o://8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b" gracePeriod=30
Sep 17 15:58:27.844484 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:27.844447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:28.292624 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:28.292582    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 15:58:28.417239 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:28.417207    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerID="8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b" exitCode=2
Sep 17 15:58:28.417409 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:28.417290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" event={"ID":"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7","Type":"ContainerDied","Data":"8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b"}
Sep 17 15:58:29.298113 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.298080    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:58:29.327256 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.327228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkw9c\" (UniqueName: \"kubernetes.io/projected/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-api-access-nkw9c\") pod \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") "
Sep 17 15:58:29.327256 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.327260    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-rbac-proxy-sar-config\") pod \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") "
Sep 17 15:58:29.327452 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.327308    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kserve-provision-location\") pod \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") "
Sep 17 15:58:29.327508 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.327486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-cabundle-cert\") pod \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") "
Sep 17 15:58:29.327561 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.327543    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-proxy-tls\") pod \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\" (UID: \"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7\") "
Sep 17 15:58:29.327846 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.327624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" (UID: "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:29.327971 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.327943    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" (UID: "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:29.328132 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.328108    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:58:29.328246 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.328136    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:58:29.329566 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.329535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-api-access-nkw9c" (OuterVolumeSpecName: "kube-api-access-nkw9c") pod "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" (UID: "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7"). InnerVolumeSpecName "kube-api-access-nkw9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:58:29.329814 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.329772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" (UID: "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:29.331839 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.331819    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" (UID: "0ea181dc-f8c3-4ed0-b6e8-db485d5196d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:29.423303 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.423221    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerID="eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f" exitCode=0
Sep 17 15:58:29.423303 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.423284    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"
Sep 17 15:58:29.423303 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.423288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" event={"ID":"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7","Type":"ContainerDied","Data":"eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f"}
Sep 17 15:58:29.423527 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.423384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v" event={"ID":"0ea181dc-f8c3-4ed0-b6e8-db485d5196d7","Type":"ContainerDied","Data":"4804f9c121f16219e4d7c594d69c1b4b5166966ab669e4f50ce3a422178ce111"}
Sep 17 15:58:29.423527 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.423434    2580 scope.go:117] "RemoveContainer" containerID="8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b"
Sep 17 15:58:29.428603 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.428582    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:58:29.428603 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.428603    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:58:29.428729 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.428613    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkw9c\" (UniqueName: \"kubernetes.io/projected/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7-kube-api-access-nkw9c\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 15:58:29.431786 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.431768    2580 scope.go:117] "RemoveContainer" containerID="eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f"
Sep 17 15:58:29.439276 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.439260    2580 scope.go:117] "RemoveContainer" containerID="e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e"
Sep 17 15:58:29.446374 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.446356    2580 scope.go:117] "RemoveContainer" containerID="8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b"
Sep 17 15:58:29.446475 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.446455    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"]
Sep 17 15:58:29.446648 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:58:29.446628    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b\": container with ID starting with 8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b not found: ID does not exist" containerID="8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b"
Sep 17 15:58:29.446716 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.446659    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b"} err="failed to get container status \"8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b\": rpc error: code = NotFound desc = could not find container \"8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b\": container with ID starting with 8a7cdd36a736abf88f7c5d0c37ca059a3e9e2ee8ad49e9bb7243626db4305c6b not found: ID does not exist"
Sep 17 15:58:29.446716 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.446685    2580 scope.go:117] "RemoveContainer" containerID="eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f"
Sep 17 15:58:29.446966 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:58:29.446950    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f\": container with ID starting with eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f not found: ID does not exist" containerID="eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f"
Sep 17 15:58:29.447019 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.446973    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f"} err="failed to get container status \"eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f\": rpc error: code = NotFound desc = could not find container \"eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f\": container with ID starting with eb188a9ad4c804c5c91c93025bbe453e281d3898faa2b7b20194182c0b30af3f not found: ID does not exist"
Sep 17 15:58:29.447019 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.446987    2580 scope.go:117] "RemoveContainer" containerID="e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e"
Sep 17 15:58:29.447200 ip-10-0-128-144 kubenswrapper[2580]: E0917 15:58:29.447185    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e\": container with ID starting with e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e not found: ID does not exist" containerID="e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e"
Sep 17 15:58:29.447238 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.447204    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e"} err="failed to get container status \"e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e\": rpc error: code = NotFound desc = could not find container \"e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e\": container with ID starting with e7028bdfe2997b0a3d268429873577ad0097aab6e45d572b4df3e3ea2913324e not found: ID does not exist"
Sep 17 15:58:29.451600 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.451584    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7j94v"]
Sep 17 15:58:29.673974 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:29.673905    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" path="/var/lib/kubelet/pods/0ea181dc-f8c3-4ed0-b6e8-db485d5196d7/volumes"
Sep 17 15:58:33.004920 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.004886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:33.033340 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033305    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"]
Sep 17 15:58:33.033581 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033569    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kube-rbac-proxy"
Sep 17 15:58:33.033581 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033582    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kube-rbac-proxy"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033590    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kube-rbac-proxy"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033596    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kube-rbac-proxy"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033605    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kserve-container"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033611    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kserve-container"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033618    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="storage-initializer"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033623    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="storage-initializer"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033633    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="storage-initializer"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033638    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="storage-initializer"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033650    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container"
Sep 17 15:58:33.033668 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container"
Sep 17 15:58:33.034008 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033701    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kube-rbac-proxy"
Sep 17 15:58:33.034008 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033709    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kserve-container"
Sep 17 15:58:33.034008 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033716    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1bf1e8-7332-47c2-b1cb-09bebe39c62a" containerName="kserve-container"
Sep 17 15:58:33.034008 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.033722    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ea181dc-f8c3-4ed0-b6e8-db485d5196d7" containerName="kube-rbac-proxy"
Sep 17 15:58:33.035637 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.035610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.037858 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.037837    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 15:58:33.037858 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.037841    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 15:58:33.038017 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.037905    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 15:58:33.038116 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.038100    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 15:58:33.038163 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.038120    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:58:33.038163 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.038136    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 15:58:33.038242 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.038193    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 15:58:33.045642 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.045619    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"]
Sep 17 15:58:33.158261 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.158228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6zt\" (UniqueName: \"kubernetes.io/projected/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-api-access-cg6zt\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.158473 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.158276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.158473 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.158304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a1b2bd-8d16-48dc-8268-1545bd48374a-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.158473 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.158372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1b2bd-8d16-48dc-8268-1545bd48374a-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.158473 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.158414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.236540 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.236506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:33.259549 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.259474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1b2bd-8d16-48dc-8268-1545bd48374a-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.259549 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.259508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.259748 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.259564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6zt\" (UniqueName: \"kubernetes.io/projected/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-api-access-cg6zt\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.259748 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.259606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.259748 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.259633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a1b2bd-8d16-48dc-8268-1545bd48374a-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.260191 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.260168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a1b2bd-8d16-48dc-8268-1545bd48374a-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.260391 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.260371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.260434 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.260393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.262123 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.262103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1b2bd-8d16-48dc-8268-1545bd48374a-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.268661 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.268634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6zt\" (UniqueName: \"kubernetes.io/projected/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-api-access-cg6zt\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8zfzx\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.346681 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.346638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:58:33.615930 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:33.615907    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"]
Sep 17 15:58:33.617726 ip-10-0-128-144 kubenswrapper[2580]: W0917 15:58:33.617700    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a1b2bd_8d16_48dc_8268_1545bd48374a.slice/crio-ebb1403aa75862d47ad74a714c03dbb5f34e2c99989cc12d3246ecff6fb6e345 WatchSource:0}: Error finding container ebb1403aa75862d47ad74a714c03dbb5f34e2c99989cc12d3246ecff6fb6e345: Status 404 returned error can't find the container with id ebb1403aa75862d47ad74a714c03dbb5f34e2c99989cc12d3246ecff6fb6e345
Sep 17 15:58:34.444930 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:34.444888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" event={"ID":"15a1b2bd-8d16-48dc-8268-1545bd48374a","Type":"ContainerStarted","Data":"8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33"}
Sep 17 15:58:34.445291 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:34.444942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" event={"ID":"15a1b2bd-8d16-48dc-8268-1545bd48374a","Type":"ContainerStarted","Data":"ebb1403aa75862d47ad74a714c03dbb5f34e2c99989cc12d3246ecff6fb6e345"}
Sep 17 15:58:35.653525 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:35.653497    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:58:35.653979 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:58:35.653724    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 15:59:03.554380 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:03.554342    2580 generic.go:356] "Generic (PLEG): container finished" podID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerID="8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33" exitCode=0
Sep 17 15:59:03.554792 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:03.554424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" event={"ID":"15a1b2bd-8d16-48dc-8268-1545bd48374a","Type":"ContainerDied","Data":"8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33"}
Sep 17 15:59:04.560186 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:04.560152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" event={"ID":"15a1b2bd-8d16-48dc-8268-1545bd48374a","Type":"ContainerStarted","Data":"a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1"}
Sep 17 15:59:04.560621 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:04.560213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" event={"ID":"15a1b2bd-8d16-48dc-8268-1545bd48374a","Type":"ContainerStarted","Data":"58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb"}
Sep 17 15:59:04.560621 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:04.560401    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:59:04.579670 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:04.579613    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" podStartSLOduration=2.264353887 podStartE2EDuration="31.579592028s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.315238141s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:04.578685016 +0000 UTC m=+1229.421047822" watchObservedRunningTime="2026-09-17 15:59:04.579592028 +0000 UTC m=+1229.421954837"
Sep 17 15:59:05.565426 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:05.565400    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:59:11.575540 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:11.575511    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 15:59:33.618951 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:33.618867    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:38.191436 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:38.191404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:38.776222 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:38.776168    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:38.919542 ip-10-0-128-144 kubenswrapper[2580]: I0917 15:59:38.919502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:16.570540 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:16.570501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:16.788026 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:16.787995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:34.703971 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:34.703932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 16:00:45.711983 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:45.711940    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 16:00:49.233938 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:49.233904    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"]
Sep 17 16:00:49.234345 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:49.234170    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="kserve-container" containerID="cri-o://58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb" gracePeriod=30
Sep 17 16:00:49.234345 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:49.234212    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="kube-rbac-proxy" containerID="cri-o://a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1" gracePeriod=30
Sep 17 16:00:49.267655 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:49.267623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:49.960616 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:49.960582    2580 generic.go:356] "Generic (PLEG): container finished" podID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerID="a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1" exitCode=2
Sep 17 16:00:49.960616 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:49.960630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" event={"ID":"15a1b2bd-8d16-48dc-8268-1545bd48374a","Type":"ContainerDied","Data":"a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1"}
Sep 17 16:00:50.578582 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.578554    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 16:00:50.605703 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.605681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a1b2bd-8d16-48dc-8268-1545bd48374a-kserve-provision-location\") pod \"15a1b2bd-8d16-48dc-8268-1545bd48374a\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") "
Sep 17 16:00:50.605881 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.605711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1b2bd-8d16-48dc-8268-1545bd48374a-proxy-tls\") pod \"15a1b2bd-8d16-48dc-8268-1545bd48374a\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") "
Sep 17 16:00:50.605881 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.605748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-cabundle-cert\") pod \"15a1b2bd-8d16-48dc-8268-1545bd48374a\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") "
Sep 17 16:00:50.605881 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.605776    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-rbac-proxy-sar-config\") pod \"15a1b2bd-8d16-48dc-8268-1545bd48374a\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") "
Sep 17 16:00:50.605881 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.605851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg6zt\" (UniqueName: \"kubernetes.io/projected/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-api-access-cg6zt\") pod \"15a1b2bd-8d16-48dc-8268-1545bd48374a\" (UID: \"15a1b2bd-8d16-48dc-8268-1545bd48374a\") "
Sep 17 16:00:50.607049 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.606207    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15a1b2bd-8d16-48dc-8268-1545bd48374a" (UID: "15a1b2bd-8d16-48dc-8268-1545bd48374a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:00:50.607049 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.606231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15a1b2bd-8d16-48dc-8268-1545bd48374a" (UID: "15a1b2bd-8d16-48dc-8268-1545bd48374a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:00:50.608627 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.608598    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-api-access-cg6zt" (OuterVolumeSpecName: "kube-api-access-cg6zt") pod "15a1b2bd-8d16-48dc-8268-1545bd48374a" (UID: "15a1b2bd-8d16-48dc-8268-1545bd48374a"). InnerVolumeSpecName "kube-api-access-cg6zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:00:50.608770 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.608602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a1b2bd-8d16-48dc-8268-1545bd48374a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15a1b2bd-8d16-48dc-8268-1545bd48374a" (UID: "15a1b2bd-8d16-48dc-8268-1545bd48374a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:00:50.611565 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.611540    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a1b2bd-8d16-48dc-8268-1545bd48374a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15a1b2bd-8d16-48dc-8268-1545bd48374a" (UID: "15a1b2bd-8d16-48dc-8268-1545bd48374a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:50.706500 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.706399    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:00:50.706500 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.706442    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:00:50.706500 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.706455    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6zt\" (UniqueName: \"kubernetes.io/projected/15a1b2bd-8d16-48dc-8268-1545bd48374a-kube-api-access-cg6zt\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:00:50.706500 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.706467    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15a1b2bd-8d16-48dc-8268-1545bd48374a-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:00:50.706500 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.706481    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1b2bd-8d16-48dc-8268-1545bd48374a-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:00:50.966393 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.966309    2580 generic.go:356] "Generic (PLEG): container finished" podID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerID="58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb" exitCode=0
Sep 17 16:00:50.966393 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.966354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" event={"ID":"15a1b2bd-8d16-48dc-8268-1545bd48374a","Type":"ContainerDied","Data":"58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb"}
Sep 17 16:00:50.966393 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.966360    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"
Sep 17 16:00:50.966614 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.966403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx" event={"ID":"15a1b2bd-8d16-48dc-8268-1545bd48374a","Type":"ContainerDied","Data":"ebb1403aa75862d47ad74a714c03dbb5f34e2c99989cc12d3246ecff6fb6e345"}
Sep 17 16:00:50.966614 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.966427    2580 scope.go:117] "RemoveContainer" containerID="a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1"
Sep 17 16:00:50.975002 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.974986    2580 scope.go:117] "RemoveContainer" containerID="58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb"
Sep 17 16:00:50.983894 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.983875    2580 scope.go:117] "RemoveContainer" containerID="8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33"
Sep 17 16:00:50.990502 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.990480    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"]
Sep 17 16:00:50.991767 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.991750    2580 scope.go:117] "RemoveContainer" containerID="a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1"
Sep 17 16:00:50.992273 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:00:50.992247    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1\": container with ID starting with a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1 not found: ID does not exist" containerID="a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1"
Sep 17 16:00:50.992363 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.992277    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1"} err="failed to get container status \"a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1\": rpc error: code = NotFound desc = could not find container \"a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1\": container with ID starting with a2c6eb96f1830fa05b22a8f9e85a333674967955c0340f301f89de1cca2712a1 not found: ID does not exist"
Sep 17 16:00:50.992363 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.992300    2580 scope.go:117] "RemoveContainer" containerID="58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb"
Sep 17 16:00:50.992619 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:00:50.992599    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb\": container with ID starting with 58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb not found: ID does not exist" containerID="58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb"
Sep 17 16:00:50.992666 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.992624    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb"} err="failed to get container status \"58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb\": rpc error: code = NotFound desc = could not find container \"58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb\": container with ID starting with 58f52872c49dcb6fb5e079ffbd56b2cbcc38666d5f74126096279e19483ffdfb not found: ID does not exist"
Sep 17 16:00:50.992666 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.992640    2580 scope.go:117] "RemoveContainer" containerID="8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33"
Sep 17 16:00:50.992914 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:00:50.992899    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33\": container with ID starting with 8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33 not found: ID does not exist" containerID="8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33"
Sep 17 16:00:50.992974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.992918    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33"} err="failed to get container status \"8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33\": rpc error: code = NotFound desc = could not find container \"8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33\": container with ID starting with 8f622d775b9db3aa1c19242cc843bd555e83f09df71b67942458e362589ddb33 not found: ID does not exist"
Sep 17 16:00:50.993285 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:50.993269    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8zfzx"]
Sep 17 16:00:51.674825 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:51.674778    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" path="/var/lib/kubelet/pods/15a1b2bd-8d16-48dc-8268-1545bd48374a/volumes"
Sep 17 16:00:54.447969 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.447935    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"]
Sep 17 16:00:54.448320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.448247    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="kube-rbac-proxy"
Sep 17 16:00:54.448320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.448260    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="kube-rbac-proxy"
Sep 17 16:00:54.448320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.448271    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="kserve-container"
Sep 17 16:00:54.448320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.448276    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="kserve-container"
Sep 17 16:00:54.448320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.448288    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="storage-initializer"
Sep 17 16:00:54.448320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.448296    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="storage-initializer"
Sep 17 16:00:54.448508 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.448338    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="kserve-container"
Sep 17 16:00:54.448508 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.448347    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a1b2bd-8d16-48dc-8268-1545bd48374a" containerName="kube-rbac-proxy"
Sep 17 16:00:54.453001 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.452982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.455490 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.455470    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:00:54.455661 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.455542    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:00:54.455661 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.455474    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:00:54.455661 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.455473    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 16:00:54.455929 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.455887    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:00:54.455929 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.455905    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 16:00:54.455929 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.455910    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:00:54.470093 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.470075    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"]
Sep 17 16:00:54.534351 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.534317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.534523 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.534355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcsmp\" (UniqueName: \"kubernetes.io/projected/74f05b68-1a80-4532-aa3d-5b805816e532-kube-api-access-xcsmp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.534523 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.534376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.534523 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.534454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f05b68-1a80-4532-aa3d-5b805816e532-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.534523 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.534494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f05b68-1a80-4532-aa3d-5b805816e532-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.635355 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.635321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.635483 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.635359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcsmp\" (UniqueName: \"kubernetes.io/projected/74f05b68-1a80-4532-aa3d-5b805816e532-kube-api-access-xcsmp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.635483 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.635386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.635483 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.635411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f05b68-1a80-4532-aa3d-5b805816e532-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.635483 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.635438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f05b68-1a80-4532-aa3d-5b805816e532-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.635665 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:00:54.635529    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 16:00:54.635665 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:00:54.635599    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f05b68-1a80-4532-aa3d-5b805816e532-proxy-tls podName:74f05b68-1a80-4532-aa3d-5b805816e532 nodeName:}" failed. No retries permitted until 2026-09-17 16:00:55.135582271 +0000 UTC m=+1339.977945055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/74f05b68-1a80-4532-aa3d-5b805816e532-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" (UID: "74f05b68-1a80-4532-aa3d-5b805816e532") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 16:00:54.635865 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.635844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f05b68-1a80-4532-aa3d-5b805816e532-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.636083 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.636068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.636119 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.636086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:54.644417 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:54.644390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcsmp\" (UniqueName: \"kubernetes.io/projected/74f05b68-1a80-4532-aa3d-5b805816e532-kube-api-access-xcsmp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:55.140993 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:55.140962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f05b68-1a80-4532-aa3d-5b805816e532-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:55.143445 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:55.143407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f05b68-1a80-4532-aa3d-5b805816e532-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:55.363481 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:55.363449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:00:55.636358 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:55.636324    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"]
Sep 17 16:00:55.639486 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:00:55.639460    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f05b68_1a80_4532_aa3d_5b805816e532.slice/crio-a818eb07d140d3a53a739651adb3a8e6fd9ea7a8f0777976b2d41aea996da233 WatchSource:0}: Error finding container a818eb07d140d3a53a739651adb3a8e6fd9ea7a8f0777976b2d41aea996da233: Status 404 returned error can't find the container with id a818eb07d140d3a53a739651adb3a8e6fd9ea7a8f0777976b2d41aea996da233
Sep 17 16:00:55.987242 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:55.987163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" event={"ID":"74f05b68-1a80-4532-aa3d-5b805816e532","Type":"ContainerStarted","Data":"af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15"}
Sep 17 16:00:55.987242 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:55.987215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" event={"ID":"74f05b68-1a80-4532-aa3d-5b805816e532","Type":"ContainerStarted","Data":"a818eb07d140d3a53a739651adb3a8e6fd9ea7a8f0777976b2d41aea996da233"}
Sep 17 16:00:56.365444 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:56.365417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:56.425272 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:00:56.425246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:08.919009 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:08.918968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:14.044480 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:14.044445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:14.542695 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:14.542663    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:16.195213 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:16.195174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:26.100703 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:26.100668    2580 generic.go:356] "Generic (PLEG): container finished" podID="74f05b68-1a80-4532-aa3d-5b805816e532" containerID="af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15" exitCode=0
Sep 17 16:01:26.101127 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:26.100738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" event={"ID":"74f05b68-1a80-4532-aa3d-5b805816e532","Type":"ContainerDied","Data":"af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15"}
Sep 17 16:01:27.107093 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:27.107062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" event={"ID":"74f05b68-1a80-4532-aa3d-5b805816e532","Type":"ContainerStarted","Data":"7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7"}
Sep 17 16:01:27.107574 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:27.107134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" event={"ID":"74f05b68-1a80-4532-aa3d-5b805816e532","Type":"ContainerStarted","Data":"67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531"}
Sep 17 16:01:27.107574 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:27.107348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:01:27.126816 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:27.126742    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" podStartSLOduration=3.540444155 podStartE2EDuration="33.126724844s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.586280689s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:27.126126249 +0000 UTC m=+1371.968489056" watchObservedRunningTime="2026-09-17 16:01:27.126724844 +0000 UTC m=+1371.969087650"
Sep 17 16:01:28.110584 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:28.110554    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:01:34.119920 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:34.119892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:01:47.970253 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:47.970215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:48.185252 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:01:48.185219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:02.704063 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:02.704028    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:03:11.447005 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:11.446975    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"]
Sep 17 16:03:11.447387 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:11.447206    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="kserve-container" containerID="cri-o://67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531" gracePeriod=30
Sep 17 16:03:11.447387 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:11.447297    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="kube-rbac-proxy" containerID="cri-o://7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7" gracePeriod=30
Sep 17 16:03:11.469017 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:11.468991    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:12.510693 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.510659    2580 generic.go:356] "Generic (PLEG): container finished" podID="74f05b68-1a80-4532-aa3d-5b805816e532" containerID="7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7" exitCode=2
Sep 17 16:03:12.511102 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.510727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" event={"ID":"74f05b68-1a80-4532-aa3d-5b805816e532","Type":"ContainerDied","Data":"7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7"}
Sep 17 16:03:12.670251 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.670215    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.27:8080/v2/models/isvc-autogluon-v2-variants/ready\": dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 16:03:12.795686 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.795664    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:03:12.814156 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.814132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcsmp\" (UniqueName: \"kubernetes.io/projected/74f05b68-1a80-4532-aa3d-5b805816e532-kube-api-access-xcsmp\") pod \"74f05b68-1a80-4532-aa3d-5b805816e532\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") "
Sep 17 16:03:12.814218 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.814171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-cabundle-cert\") pod \"74f05b68-1a80-4532-aa3d-5b805816e532\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") "
Sep 17 16:03:12.814218 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.814199    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f05b68-1a80-4532-aa3d-5b805816e532-proxy-tls\") pod \"74f05b68-1a80-4532-aa3d-5b805816e532\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") "
Sep 17 16:03:12.814287 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.814230    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f05b68-1a80-4532-aa3d-5b805816e532-kserve-provision-location\") pod \"74f05b68-1a80-4532-aa3d-5b805816e532\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") "
Sep 17 16:03:12.814287 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.814282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-kube-rbac-proxy-sar-config\") pod \"74f05b68-1a80-4532-aa3d-5b805816e532\" (UID: \"74f05b68-1a80-4532-aa3d-5b805816e532\") "
Sep 17 16:03:12.814623 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.814598    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74f05b68-1a80-4532-aa3d-5b805816e532" (UID: "74f05b68-1a80-4532-aa3d-5b805816e532"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:12.814732 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.814709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74f05b68-1a80-4532-aa3d-5b805816e532" (UID: "74f05b68-1a80-4532-aa3d-5b805816e532"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:12.816448 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.816421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f05b68-1a80-4532-aa3d-5b805816e532-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74f05b68-1a80-4532-aa3d-5b805816e532" (UID: "74f05b68-1a80-4532-aa3d-5b805816e532"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:12.816448 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.816437    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f05b68-1a80-4532-aa3d-5b805816e532-kube-api-access-xcsmp" (OuterVolumeSpecName: "kube-api-access-xcsmp") pod "74f05b68-1a80-4532-aa3d-5b805816e532" (UID: "74f05b68-1a80-4532-aa3d-5b805816e532"). InnerVolumeSpecName "kube-api-access-xcsmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:12.818372 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.818348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f05b68-1a80-4532-aa3d-5b805816e532-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74f05b68-1a80-4532-aa3d-5b805816e532" (UID: "74f05b68-1a80-4532-aa3d-5b805816e532"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:03:12.914826 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.914770    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:03:12.914826 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.914822    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcsmp\" (UniqueName: \"kubernetes.io/projected/74f05b68-1a80-4532-aa3d-5b805816e532-kube-api-access-xcsmp\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:03:12.915006 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.914838    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f05b68-1a80-4532-aa3d-5b805816e532-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:03:12.915006 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.914851    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f05b68-1a80-4532-aa3d-5b805816e532-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:03:12.915006 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:12.914862    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f05b68-1a80-4532-aa3d-5b805816e532-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:03:13.516736 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.516705    2580 generic.go:356] "Generic (PLEG): container finished" podID="74f05b68-1a80-4532-aa3d-5b805816e532" containerID="67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531" exitCode=0
Sep 17 16:03:13.517170 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.516754    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"
Sep 17 16:03:13.517170 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.516791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" event={"ID":"74f05b68-1a80-4532-aa3d-5b805816e532","Type":"ContainerDied","Data":"67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531"}
Sep 17 16:03:13.517170 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.516871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq" event={"ID":"74f05b68-1a80-4532-aa3d-5b805816e532","Type":"ContainerDied","Data":"a818eb07d140d3a53a739651adb3a8e6fd9ea7a8f0777976b2d41aea996da233"}
Sep 17 16:03:13.517170 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.516898    2580 scope.go:117] "RemoveContainer" containerID="7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7"
Sep 17 16:03:13.527769 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.527752    2580 scope.go:117] "RemoveContainer" containerID="67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531"
Sep 17 16:03:13.534972 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.534953    2580 scope.go:117] "RemoveContainer" containerID="af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15"
Sep 17 16:03:13.542482 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.542464    2580 scope.go:117] "RemoveContainer" containerID="7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7"
Sep 17 16:03:13.542755 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:03:13.542736    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7\": container with ID starting with 7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7 not found: ID does not exist" containerID="7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7"
Sep 17 16:03:13.542887 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.542767    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7"} err="failed to get container status \"7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7\": rpc error: code = NotFound desc = could not find container \"7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7\": container with ID starting with 7cfb86ad65c2f4b11a402405a3345500b7eb40c193ee2275d9833136528285b7 not found: ID does not exist"
Sep 17 16:03:13.542887 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.542793    2580 scope.go:117] "RemoveContainer" containerID="67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531"
Sep 17 16:03:13.543094 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:03:13.543078    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531\": container with ID starting with 67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531 not found: ID does not exist" containerID="67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531"
Sep 17 16:03:13.543157 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.543101    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531"} err="failed to get container status \"67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531\": rpc error: code = NotFound desc = could not find container \"67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531\": container with ID starting with 67e0697968350c620be5a97966f1104981806cecbc0e198291fc23472cf5c531 not found: ID does not exist"
Sep 17 16:03:13.543157 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.543121    2580 scope.go:117] "RemoveContainer" containerID="af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15"
Sep 17 16:03:13.543419 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:03:13.543402    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15\": container with ID starting with af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15 not found: ID does not exist" containerID="af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15"
Sep 17 16:03:13.543465 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.543425    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15"} err="failed to get container status \"af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15\": rpc error: code = NotFound desc = could not find container \"af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15\": container with ID starting with af30b4a00afe2509949981b048c532d2a4a2defd0151124ffb0973a89cbf4b15 not found: ID does not exist"
Sep 17 16:03:13.543513 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.543472    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"]
Sep 17 16:03:13.545775 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.545746    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hpxdq"]
Sep 17 16:03:13.674237 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:13.674208    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" path="/var/lib/kubelet/pods/74f05b68-1a80-4532-aa3d-5b805816e532/volumes"
Sep 17 16:03:16.596712 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.596680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:16.637611 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.637583    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"]
Sep 17 16:03:16.637873 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.637861    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="storage-initializer"
Sep 17 16:03:16.637942 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.637875    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="storage-initializer"
Sep 17 16:03:16.637942 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.637889    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="kube-rbac-proxy"
Sep 17 16:03:16.637942 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.637894    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="kube-rbac-proxy"
Sep 17 16:03:16.637942 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.637902    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="kserve-container"
Sep 17 16:03:16.637942 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.637909    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="kserve-container"
Sep 17 16:03:16.638108 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.637957    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="kserve-container"
Sep 17 16:03:16.638108 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.637966    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f05b68-1a80-4532-aa3d-5b805816e532" containerName="kube-rbac-proxy"
Sep 17 16:03:16.642745 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.642726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.644920 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.644895    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 16:03:16.644920 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.644907    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:03:16.645088 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.645016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 16:03:16.645172 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.645157    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:03:16.645500 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.645483    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:03:16.645579 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.645486    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:03:16.645579 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.645562    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:03:16.653499 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.653473    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"]
Sep 17 16:03:16.740303 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.740260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.740303 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.740312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2js9\" (UniqueName: \"kubernetes.io/projected/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-api-access-h2js9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.740543 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.740363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1e0465-32db-4e93-95a6-b94bcb633433-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.740543 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.740426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1e0465-32db-4e93-95a6-b94bcb633433-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.740543 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.740475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.837590 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.837561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:16.841548 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.841524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1e0465-32db-4e93-95a6-b94bcb633433-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.841637 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.841555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.841637 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.841598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.841742 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.841636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2js9\" (UniqueName: \"kubernetes.io/projected/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-api-access-h2js9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.841742 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.841670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1e0465-32db-4e93-95a6-b94bcb633433-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.842088 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.842069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1e0465-32db-4e93-95a6-b94bcb633433-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.842280 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.842262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.842354 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.842334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.844055 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.844041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1e0465-32db-4e93-95a6-b94bcb633433-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.852079 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.852024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2js9\" (UniqueName: \"kubernetes.io/projected/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-api-access-h2js9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:16.953330 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:16.953301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:17.227182 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:17.227072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"]
Sep 17 16:03:17.229301 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:03:17.229276    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1e0465_32db_4e93_95a6_b94bcb633433.slice/crio-d2e40da7c47acc3f7bb647c6e2c197acec25175f06523d8fa4d8c03cc811f219 WatchSource:0}: Error finding container d2e40da7c47acc3f7bb647c6e2c197acec25175f06523d8fa4d8c03cc811f219: Status 404 returned error can't find the container with id d2e40da7c47acc3f7bb647c6e2c197acec25175f06523d8fa4d8c03cc811f219
Sep 17 16:03:17.535607 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:17.535523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" event={"ID":"9d1e0465-32db-4e93-95a6-b94bcb633433","Type":"ContainerStarted","Data":"175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1"}
Sep 17 16:03:17.535607 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:17.535581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" event={"ID":"9d1e0465-32db-4e93-95a6-b94bcb633433","Type":"ContainerStarted","Data":"d2e40da7c47acc3f7bb647c6e2c197acec25175f06523d8fa4d8c03cc811f219"}
Sep 17 16:03:18.111745 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:18.111705    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:23.299975 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:23.299940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:23.451985 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:23.451929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:35.677341 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:35.677316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:03:35.678286 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:35.678266    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:03:38.614544 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:38.614508    2580 generic.go:356] "Generic (PLEG): container finished" podID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerID="175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1" exitCode=0
Sep 17 16:03:38.615042 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:38.614577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" event={"ID":"9d1e0465-32db-4e93-95a6-b94bcb633433","Type":"ContainerDied","Data":"175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1"}
Sep 17 16:03:39.621210 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:39.621167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" event={"ID":"9d1e0465-32db-4e93-95a6-b94bcb633433","Type":"ContainerStarted","Data":"413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d"}
Sep 17 16:03:39.621672 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:39.621288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" event={"ID":"9d1e0465-32db-4e93-95a6-b94bcb633433","Type":"ContainerStarted","Data":"4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0"}
Sep 17 16:03:39.621672 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:39.621577    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:39.640995 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:39.640949    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" podStartSLOduration=2.866385146 podStartE2EDuration="23.640936584s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.774551438s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:03:39.640277697 +0000 UTC m=+1504.482640502" watchObservedRunningTime="2026-09-17 16:03:39.640936584 +0000 UTC m=+1504.483299388"
Sep 17 16:03:40.627130 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:40.627098    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:03:46.636260 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:03:46.636226    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:05:15.706844 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:15.706780    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:05:18.174461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:18.174431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:18.260040 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:18.260014    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:23.147990 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:23.147960    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"]
Sep 17 16:05:23.148489 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:23.148200    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="kserve-container" containerID="cri-o://4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0" gracePeriod=30
Sep 17 16:05:23.148489 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:23.148261    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="kube-rbac-proxy" containerID="cri-o://413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d" gracePeriod=30
Sep 17 16:05:23.167693 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:23.167670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:24.015329 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.015291    2580 generic.go:356] "Generic (PLEG): container finished" podID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerID="413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d" exitCode=2
Sep 17 16:05:24.015546 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.015362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" event={"ID":"9d1e0465-32db-4e93-95a6-b94bcb633433","Type":"ContainerDied","Data":"413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d"}
Sep 17 16:05:24.485044 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.485018    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:05:24.519050 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.519026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1e0465-32db-4e93-95a6-b94bcb633433-kserve-provision-location\") pod \"9d1e0465-32db-4e93-95a6-b94bcb633433\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") "
Sep 17 16:05:24.519187 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.519059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-rbac-proxy-sar-config\") pod \"9d1e0465-32db-4e93-95a6-b94bcb633433\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") "
Sep 17 16:05:24.519187 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.519141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-cabundle-cert\") pod \"9d1e0465-32db-4e93-95a6-b94bcb633433\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") "
Sep 17 16:05:24.519187 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.519173    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2js9\" (UniqueName: \"kubernetes.io/projected/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-api-access-h2js9\") pod \"9d1e0465-32db-4e93-95a6-b94bcb633433\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") "
Sep 17 16:05:24.519334 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.519206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1e0465-32db-4e93-95a6-b94bcb633433-proxy-tls\") pod \"9d1e0465-32db-4e93-95a6-b94bcb633433\" (UID: \"9d1e0465-32db-4e93-95a6-b94bcb633433\") "
Sep 17 16:05:24.519528 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.519489    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d1e0465-32db-4e93-95a6-b94bcb633433" (UID: "9d1e0465-32db-4e93-95a6-b94bcb633433"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:05:24.519615 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.519574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d1e0465-32db-4e93-95a6-b94bcb633433" (UID: "9d1e0465-32db-4e93-95a6-b94bcb633433"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:05:24.521512 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.521469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-api-access-h2js9" (OuterVolumeSpecName: "kube-api-access-h2js9") pod "9d1e0465-32db-4e93-95a6-b94bcb633433" (UID: "9d1e0465-32db-4e93-95a6-b94bcb633433"). InnerVolumeSpecName "kube-api-access-h2js9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:05:24.521616 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.521580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1e0465-32db-4e93-95a6-b94bcb633433-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d1e0465-32db-4e93-95a6-b94bcb633433" (UID: "9d1e0465-32db-4e93-95a6-b94bcb633433"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:05:24.524444 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.524411    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1e0465-32db-4e93-95a6-b94bcb633433-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d1e0465-32db-4e93-95a6-b94bcb633433" (UID: "9d1e0465-32db-4e93-95a6-b94bcb633433"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:05:24.619667 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.619641    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:05:24.619667 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.619666    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2js9\" (UniqueName: \"kubernetes.io/projected/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-api-access-h2js9\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:05:24.619868 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.619678    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1e0465-32db-4e93-95a6-b94bcb633433-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:05:24.619868 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.619686    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1e0465-32db-4e93-95a6-b94bcb633433-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:05:24.619868 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:24.619695    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1e0465-32db-4e93-95a6-b94bcb633433-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:05:25.021215 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.021135    2580 generic.go:356] "Generic (PLEG): container finished" podID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerID="4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0" exitCode=0
Sep 17 16:05:25.021215 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.021186    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"
Sep 17 16:05:25.021215 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.021204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" event={"ID":"9d1e0465-32db-4e93-95a6-b94bcb633433","Type":"ContainerDied","Data":"4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0"}
Sep 17 16:05:25.021415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.021277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht" event={"ID":"9d1e0465-32db-4e93-95a6-b94bcb633433","Type":"ContainerDied","Data":"d2e40da7c47acc3f7bb647c6e2c197acec25175f06523d8fa4d8c03cc811f219"}
Sep 17 16:05:25.021415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.021316    2580 scope.go:117] "RemoveContainer" containerID="413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d"
Sep 17 16:05:25.030660 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.030642    2580 scope.go:117] "RemoveContainer" containerID="4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0"
Sep 17 16:05:25.039765 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.039700    2580 scope.go:117] "RemoveContainer" containerID="175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1"
Sep 17 16:05:25.047207 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.047190    2580 scope.go:117] "RemoveContainer" containerID="413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d"
Sep 17 16:05:25.047529 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:05:25.047509    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d\": container with ID starting with 413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d not found: ID does not exist" containerID="413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d"
Sep 17 16:05:25.047579 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.047538    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d"} err="failed to get container status \"413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d\": rpc error: code = NotFound desc = could not find container \"413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d\": container with ID starting with 413224804a0e924ff10f5c7d501541b0b9d7e54f9e205f4b879ffd917346631d not found: ID does not exist"
Sep 17 16:05:25.047579 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.047556    2580 scope.go:117] "RemoveContainer" containerID="4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0"
Sep 17 16:05:25.047840 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:05:25.047825    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0\": container with ID starting with 4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0 not found: ID does not exist" containerID="4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0"
Sep 17 16:05:25.047900 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.047845    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0"} err="failed to get container status \"4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0\": rpc error: code = NotFound desc = could not find container \"4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0\": container with ID starting with 4264cebbf9ecdce6bae63030d6e24d081edf804a5bf9d0ccd664a7516c368bd0 not found: ID does not exist"
Sep 17 16:05:25.047900 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.047857    2580 scope.go:117] "RemoveContainer" containerID="175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1"
Sep 17 16:05:25.048094 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:05:25.048078    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1\": container with ID starting with 175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1 not found: ID does not exist" containerID="175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1"
Sep 17 16:05:25.048524 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.048098    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1"} err="failed to get container status \"175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1\": rpc error: code = NotFound desc = could not find container \"175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1\": container with ID starting with 175defab217a1a2c9ec8d43c35314572bedf1e787f54191e275ba69da716dfc1 not found: ID does not exist"
Sep 17 16:05:25.053884 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.053864    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"]
Sep 17 16:05:25.061552 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.061530    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-26xht"]
Sep 17 16:05:25.674168 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:25.674139    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" path="/var/lib/kubelet/pods/9d1e0465-32db-4e93-95a6-b94bcb633433/volumes"
Sep 17 16:05:28.282328 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.282251    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:28.356139 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.356113    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"]
Sep 17 16:05:28.356410 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.356399    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="storage-initializer"
Sep 17 16:05:28.356450 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.356412    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="storage-initializer"
Sep 17 16:05:28.356450 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.356427    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="kserve-container"
Sep 17 16:05:28.356450 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.356432    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="kserve-container"
Sep 17 16:05:28.356450 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.356442    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="kube-rbac-proxy"
Sep 17 16:05:28.356450 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.356448    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="kube-rbac-proxy"
Sep 17 16:05:28.356595 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.356492    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="kserve-container"
Sep 17 16:05:28.356595 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.356500    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1e0465-32db-4e93-95a6-b94bcb633433" containerName="kube-rbac-proxy"
Sep 17 16:05:28.360777 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.360758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.364905 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.364886    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:05:28.365011 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.364995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:05:28.365238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.365210    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:05:28.365434 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.365421    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:05:28.365613 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.365597    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:05:28.365708 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.365689    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 16:05:28.365787 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.365737    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 16:05:28.371587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.371568    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"]
Sep 17 16:05:28.448530 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.448502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.448664 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.448553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6ln\" (UniqueName: \"kubernetes.io/projected/2635663b-66cb-455c-ba65-94f9769833be-kube-api-access-6l6ln\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.448664 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.448611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.448664 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.448636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635663b-66cb-455c-ba65-94f9769833be-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.448775 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.448704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635663b-66cb-455c-ba65-94f9769833be-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.549588 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.549510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6ln\" (UniqueName: \"kubernetes.io/projected/2635663b-66cb-455c-ba65-94f9769833be-kube-api-access-6l6ln\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.549588 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.549556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.549588 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.549582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635663b-66cb-455c-ba65-94f9769833be-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.549881 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.549621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635663b-66cb-455c-ba65-94f9769833be-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.549881 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.549640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.550093 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.550069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635663b-66cb-455c-ba65-94f9769833be-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.550381 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.550364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.550431 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.550420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.552153 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.552130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635663b-66cb-455c-ba65-94f9769833be-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.564634 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.564610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6ln\" (UniqueName: \"kubernetes.io/projected/2635663b-66cb-455c-ba65-94f9769833be-kube-api-access-6l6ln\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.670016 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.669981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:28.717719 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.717684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:05:28.949516 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:28.949488    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"]
Sep 17 16:05:28.952491 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:05:28.952463    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2635663b_66cb_455c_ba65_94f9769833be.slice/crio-59091a1d2e278ed1805a811eb00c0585cf006291baaa532c739425dcf1395a20 WatchSource:0}: Error finding container 59091a1d2e278ed1805a811eb00c0585cf006291baaa532c739425dcf1395a20: Status 404 returned error can't find the container with id 59091a1d2e278ed1805a811eb00c0585cf006291baaa532c739425dcf1395a20
Sep 17 16:05:29.037556 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:29.037520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" event={"ID":"2635663b-66cb-455c-ba65-94f9769833be","Type":"ContainerStarted","Data":"68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1"}
Sep 17 16:05:29.037699 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:29.037598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" event={"ID":"2635663b-66cb-455c-ba65-94f9769833be","Type":"ContainerStarted","Data":"59091a1d2e278ed1805a811eb00c0585cf006291baaa532c739425dcf1395a20"}
Sep 17 16:05:39.074245 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:39.074149    2580 generic.go:356] "Generic (PLEG): container finished" podID="2635663b-66cb-455c-ba65-94f9769833be" containerID="68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1" exitCode=0
Sep 17 16:05:39.074245 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:39.074212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" event={"ID":"2635663b-66cb-455c-ba65-94f9769833be","Type":"ContainerDied","Data":"68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1"}
Sep 17 16:05:40.083560 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:40.083521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" event={"ID":"2635663b-66cb-455c-ba65-94f9769833be","Type":"ContainerStarted","Data":"6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175"}
Sep 17 16:05:40.083967 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:40.083583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" event={"ID":"2635663b-66cb-455c-ba65-94f9769833be","Type":"ContainerStarted","Data":"998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17"}
Sep 17 16:05:40.083967 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:40.083831    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:40.084075 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:40.083992    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:40.085146 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:40.085115    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:05:40.105438 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:40.105389    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" podStartSLOduration=2.370741019 podStartE2EDuration="12.105369408s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.734628389s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:05:40.104060249 +0000 UTC m=+1624.946423054" watchObservedRunningTime="2026-09-17 16:05:40.105369408 +0000 UTC m=+1624.947732214"
Sep 17 16:05:41.088151 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:41.088101    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:05:46.093008 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:46.092976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:05:46.093537 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:46.093510    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:05:56.094771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:05:56.094747    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:06:10.325843 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:10.325788    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"]
Sep 17 16:06:10.326371 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:10.326054    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kserve-container" containerID="cri-o://998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17" gracePeriod=30
Sep 17 16:06:10.326371 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:10.326096    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kube-rbac-proxy" containerID="cri-o://6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175" gracePeriod=30
Sep 17 16:06:10.377988 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:10.377956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:11.088285 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:11.088238    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 16:06:11.233607 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:11.233578    2580 generic.go:356] "Generic (PLEG): container finished" podID="2635663b-66cb-455c-ba65-94f9769833be" containerID="6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175" exitCode=2
Sep 17 16:06:11.233833 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:11.233657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" event={"ID":"2635663b-66cb-455c-ba65-94f9769833be","Type":"ContainerDied","Data":"6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175"}
Sep 17 16:06:11.960018 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:11.959994    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:06:12.004703 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.004678    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l6ln\" (UniqueName: \"kubernetes.io/projected/2635663b-66cb-455c-ba65-94f9769833be-kube-api-access-6l6ln\") pod \"2635663b-66cb-455c-ba65-94f9769833be\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") "
Sep 17 16:06:12.004862 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.004737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635663b-66cb-455c-ba65-94f9769833be-proxy-tls\") pod \"2635663b-66cb-455c-ba65-94f9769833be\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") "
Sep 17 16:06:12.004862 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.004767    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-cabundle-cert\") pod \"2635663b-66cb-455c-ba65-94f9769833be\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") "
Sep 17 16:06:12.004862 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.004810    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635663b-66cb-455c-ba65-94f9769833be-kserve-provision-location\") pod \"2635663b-66cb-455c-ba65-94f9769833be\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") "
Sep 17 16:06:12.004862 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.004835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-kube-rbac-proxy-sar-config\") pod \"2635663b-66cb-455c-ba65-94f9769833be\" (UID: \"2635663b-66cb-455c-ba65-94f9769833be\") "
Sep 17 16:06:12.005169 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.005143    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2635663b-66cb-455c-ba65-94f9769833be" (UID: "2635663b-66cb-455c-ba65-94f9769833be"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:06:12.005287 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.005242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2635663b-66cb-455c-ba65-94f9769833be" (UID: "2635663b-66cb-455c-ba65-94f9769833be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:06:12.005730 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.005707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2635663b-66cb-455c-ba65-94f9769833be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2635663b-66cb-455c-ba65-94f9769833be" (UID: "2635663b-66cb-455c-ba65-94f9769833be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:06:12.006669 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.006645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2635663b-66cb-455c-ba65-94f9769833be-kube-api-access-6l6ln" (OuterVolumeSpecName: "kube-api-access-6l6ln") pod "2635663b-66cb-455c-ba65-94f9769833be" (UID: "2635663b-66cb-455c-ba65-94f9769833be"). InnerVolumeSpecName "kube-api-access-6l6ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:06:12.006752 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.006725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2635663b-66cb-455c-ba65-94f9769833be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2635663b-66cb-455c-ba65-94f9769833be" (UID: "2635663b-66cb-455c-ba65-94f9769833be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:06:12.105376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.105309    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l6ln\" (UniqueName: \"kubernetes.io/projected/2635663b-66cb-455c-ba65-94f9769833be-kube-api-access-6l6ln\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:12.105376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.105332    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2635663b-66cb-455c-ba65-94f9769833be-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:12.105376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.105341    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:12.105376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.105349    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2635663b-66cb-455c-ba65-94f9769833be-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:12.105376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.105358    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2635663b-66cb-455c-ba65-94f9769833be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:12.240045 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.240014    2580 generic.go:356] "Generic (PLEG): container finished" podID="2635663b-66cb-455c-ba65-94f9769833be" containerID="998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17" exitCode=0
Sep 17 16:06:12.240188 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.240064    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"
Sep 17 16:06:12.240188 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.240060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" event={"ID":"2635663b-66cb-455c-ba65-94f9769833be","Type":"ContainerDied","Data":"998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17"}
Sep 17 16:06:12.240273 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.240195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk" event={"ID":"2635663b-66cb-455c-ba65-94f9769833be","Type":"ContainerDied","Data":"59091a1d2e278ed1805a811eb00c0585cf006291baaa532c739425dcf1395a20"}
Sep 17 16:06:12.240273 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.240226    2580 scope.go:117] "RemoveContainer" containerID="6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175"
Sep 17 16:06:12.248860 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.248842    2580 scope.go:117] "RemoveContainer" containerID="998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17"
Sep 17 16:06:12.255974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.255958    2580 scope.go:117] "RemoveContainer" containerID="68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1"
Sep 17 16:06:12.262478 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.262456    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"]
Sep 17 16:06:12.263777 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.263761    2580 scope.go:117] "RemoveContainer" containerID="6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175"
Sep 17 16:06:12.264276 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:06:12.264248    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175\": container with ID starting with 6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175 not found: ID does not exist" containerID="6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175"
Sep 17 16:06:12.264376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.264286    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175"} err="failed to get container status \"6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175\": rpc error: code = NotFound desc = could not find container \"6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175\": container with ID starting with 6fe111f4489e274b284d1634a573d505dd3d0b50fc30a0668f21c42b3ff01175 not found: ID does not exist"
Sep 17 16:06:12.264376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.264308    2580 scope.go:117] "RemoveContainer" containerID="998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17"
Sep 17 16:06:12.264619 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:06:12.264599    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17\": container with ID starting with 998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17 not found: ID does not exist" containerID="998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17"
Sep 17 16:06:12.264680 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.264627    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17"} err="failed to get container status \"998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17\": rpc error: code = NotFound desc = could not find container \"998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17\": container with ID starting with 998c7ac5ea2157632131339852ae5f29f1e43cf4a1ee90c7f4178cb3355c9e17 not found: ID does not exist"
Sep 17 16:06:12.264680 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.264642    2580 scope.go:117] "RemoveContainer" containerID="68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1"
Sep 17 16:06:12.265073 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:06:12.265054    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1\": container with ID starting with 68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1 not found: ID does not exist" containerID="68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1"
Sep 17 16:06:12.265126 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.265079    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1"} err="failed to get container status \"68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1\": rpc error: code = NotFound desc = could not find container \"68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1\": container with ID starting with 68dd486c8cc213eb8ae547d52ed6bd2fa32038a7810c3401532e2a0a39e5f4b1 not found: ID does not exist"
Sep 17 16:06:12.265910 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:12.265893    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-tv5lk"]
Sep 17 16:06:13.674548 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:13.674506    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2635663b-66cb-455c-ba65-94f9769833be" path="/var/lib/kubelet/pods/2635663b-66cb-455c-ba65-94f9769833be/volumes"
Sep 17 16:06:15.490378 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.490337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:15.529408 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.529375    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"]
Sep 17 16:06:15.529655 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.529643    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="storage-initializer"
Sep 17 16:06:15.529700 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.529658    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="storage-initializer"
Sep 17 16:06:15.529700 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.529670    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kube-rbac-proxy"
Sep 17 16:06:15.529700 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.529676    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kube-rbac-proxy"
Sep 17 16:06:15.529700 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.529695    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kserve-container"
Sep 17 16:06:15.529853 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.529702    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kserve-container"
Sep 17 16:06:15.529853 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.529748    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kserve-container"
Sep 17 16:06:15.529853 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.529759    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2635663b-66cb-455c-ba65-94f9769833be" containerName="kube-rbac-proxy"
Sep 17 16:06:15.534456 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.534438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.536864 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.536842    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 16:06:15.536980 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.536873    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:06:15.536980 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.536848    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:06:15.536980 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.536847    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:06:15.537139 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.537102    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:06:15.537245 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.537232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 16:06:15.537288 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.537272    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:06:15.544469 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.544451    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"]
Sep 17 16:06:15.627197 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.627164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.627197 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.627199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c675c3c2-4894-4cbe-88c2-e8136f0090e8-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.627384 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.627232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.627384 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.627281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.627384 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.627326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zl2d\" (UniqueName: \"kubernetes.io/projected/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-api-access-5zl2d\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.647961 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.647936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:15.727701 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.727671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.727866 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.727716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.727866 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.727767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zl2d\" (UniqueName: \"kubernetes.io/projected/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-api-access-5zl2d\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.727866 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.727826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.727866 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.727854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c675c3c2-4894-4cbe-88c2-e8136f0090e8-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.728197 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.728175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.728335 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.728318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.728400 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.728384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.730472 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.730452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c675c3c2-4894-4cbe-88c2-e8136f0090e8-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.738854 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.738835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zl2d\" (UniqueName: \"kubernetes.io/projected/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-api-access-5zl2d\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:15.845018 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:15.844926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:16.114847 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:16.114823    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"]
Sep 17 16:06:16.117066 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:06:16.117034    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc675c3c2_4894_4cbe_88c2_e8136f0090e8.slice/crio-5812fdb59ad805abc6ef1bad3420c585eef3f1656526b38033465465692edd65 WatchSource:0}: Error finding container 5812fdb59ad805abc6ef1bad3420c585eef3f1656526b38033465465692edd65: Status 404 returned error can't find the container with id 5812fdb59ad805abc6ef1bad3420c585eef3f1656526b38033465465692edd65
Sep 17 16:06:16.264688 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:16.264656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" event={"ID":"c675c3c2-4894-4cbe-88c2-e8136f0090e8","Type":"ContainerStarted","Data":"1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7"}
Sep 17 16:06:16.264871 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:16.264733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" event={"ID":"c675c3c2-4894-4cbe-88c2-e8136f0090e8","Type":"ContainerStarted","Data":"5812fdb59ad805abc6ef1bad3420c585eef3f1656526b38033465465692edd65"}
Sep 17 16:06:17.229153 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:17.229119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:24.294652 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:24.294620    2580 generic.go:356] "Generic (PLEG): container finished" podID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerID="1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7" exitCode=0
Sep 17 16:06:24.295069 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:24.294696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" event={"ID":"c675c3c2-4894-4cbe-88c2-e8136f0090e8","Type":"ContainerDied","Data":"1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7"}
Sep 17 16:06:25.300117 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:25.300084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" event={"ID":"c675c3c2-4894-4cbe-88c2-e8136f0090e8","Type":"ContainerStarted","Data":"bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf"}
Sep 17 16:06:25.300514 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:25.300149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" event={"ID":"c675c3c2-4894-4cbe-88c2-e8136f0090e8","Type":"ContainerStarted","Data":"945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6"}
Sep 17 16:06:25.300514 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:25.300418    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:25.300650 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:25.300586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:25.302066 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:25.302039    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:06:25.321842 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:25.321760    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" podStartSLOduration=3.087987102 podStartE2EDuration="10.321742546s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.233755444s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:25.319962096 +0000 UTC m=+1670.162324900" watchObservedRunningTime="2026-09-17 16:06:25.321742546 +0000 UTC m=+1670.164105351"
Sep 17 16:06:26.304323 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:26.304286    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:06:27.307690 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:27.307648    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:06:32.311769 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:32.311744    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:32.312202 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:32.312178    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:06:33.460704 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:33.460676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:38.649559 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:38.649528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:38.727341 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:38.727309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:42.312969 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:42.312942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:50.873975 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:50.873944    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"]
Sep 17 16:06:50.874443 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:50.874177    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kserve-container" containerID="cri-o://945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6" gracePeriod=30
Sep 17 16:06:50.874443 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:50.874220    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kube-rbac-proxy" containerID="cri-o://bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf" gracePeriod=30
Sep 17 16:06:50.888823 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:50.888754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:51.401463 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:51.401425    2580 generic.go:356] "Generic (PLEG): container finished" podID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerID="bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf" exitCode=2
Sep 17 16:06:51.401669 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:51.401473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" event={"ID":"c675c3c2-4894-4cbe-88c2-e8136f0090e8","Type":"ContainerDied","Data":"bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf"}
Sep 17 16:06:52.313951 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.313927    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:52.408731 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.408692    2580 generic.go:356] "Generic (PLEG): container finished" podID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerID="945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6" exitCode=0
Sep 17 16:06:52.408954 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.408761    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"
Sep 17 16:06:52.408954 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.408779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" event={"ID":"c675c3c2-4894-4cbe-88c2-e8136f0090e8","Type":"ContainerDied","Data":"945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6"}
Sep 17 16:06:52.408954 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.408862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" event={"ID":"c675c3c2-4894-4cbe-88c2-e8136f0090e8","Type":"ContainerDied","Data":"5812fdb59ad805abc6ef1bad3420c585eef3f1656526b38033465465692edd65"}
Sep 17 16:06:52.408954 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.408892    2580 scope.go:117] "RemoveContainer" containerID="bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf"
Sep 17 16:06:52.415380 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.415360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zl2d\" (UniqueName: \"kubernetes.io/projected/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-api-access-5zl2d\") pod \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") "
Sep 17 16:06:52.415489 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.415445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-rbac-proxy-sar-config\") pod \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") "
Sep 17 16:06:52.415551 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.415536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c675c3c2-4894-4cbe-88c2-e8136f0090e8-proxy-tls\") pod \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") "
Sep 17 16:06:52.415601 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.415582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kserve-provision-location\") pod \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") "
Sep 17 16:06:52.415662 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.415617    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-cabundle-cert\") pod \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\" (UID: \"c675c3c2-4894-4cbe-88c2-e8136f0090e8\") "
Sep 17 16:06:52.416140 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.416108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c675c3c2-4894-4cbe-88c2-e8136f0090e8" (UID: "c675c3c2-4894-4cbe-88c2-e8136f0090e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:06:52.416140 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.416121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c675c3c2-4894-4cbe-88c2-e8136f0090e8" (UID: "c675c3c2-4894-4cbe-88c2-e8136f0090e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:06:52.416840 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.416814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c675c3c2-4894-4cbe-88c2-e8136f0090e8" (UID: "c675c3c2-4894-4cbe-88c2-e8136f0090e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:06:52.417505 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.417484    2580 scope.go:117] "RemoveContainer" containerID="945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6"
Sep 17 16:06:52.417862 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.417841    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c675c3c2-4894-4cbe-88c2-e8136f0090e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c675c3c2-4894-4cbe-88c2-e8136f0090e8" (UID: "c675c3c2-4894-4cbe-88c2-e8136f0090e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:06:52.417974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.417954    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-api-access-5zl2d" (OuterVolumeSpecName: "kube-api-access-5zl2d") pod "c675c3c2-4894-4cbe-88c2-e8136f0090e8" (UID: "c675c3c2-4894-4cbe-88c2-e8136f0090e8"). InnerVolumeSpecName "kube-api-access-5zl2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:06:52.433453 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.433435    2580 scope.go:117] "RemoveContainer" containerID="1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7"
Sep 17 16:06:52.440516 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.440501    2580 scope.go:117] "RemoveContainer" containerID="bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf"
Sep 17 16:06:52.440780 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:06:52.440760    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf\": container with ID starting with bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf not found: ID does not exist" containerID="bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf"
Sep 17 16:06:52.440887 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.440791    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf"} err="failed to get container status \"bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf\": rpc error: code = NotFound desc = could not find container \"bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf\": container with ID starting with bffb057a38f3fee669c9e19fe193a51532960dcf06fab041f6280b48b67af8bf not found: ID does not exist"
Sep 17 16:06:52.440887 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.440832    2580 scope.go:117] "RemoveContainer" containerID="945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6"
Sep 17 16:06:52.441097 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:06:52.441081    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6\": container with ID starting with 945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6 not found: ID does not exist" containerID="945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6"
Sep 17 16:06:52.441140 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.441103    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6"} err="failed to get container status \"945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6\": rpc error: code = NotFound desc = could not find container \"945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6\": container with ID starting with 945bf84b516bf2132fa39f722af158ad86ee5d9995692473af2e2fcf628273b6 not found: ID does not exist"
Sep 17 16:06:52.441140 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.441118    2580 scope.go:117] "RemoveContainer" containerID="1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7"
Sep 17 16:06:52.441385 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:06:52.441366    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7\": container with ID starting with 1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7 not found: ID does not exist" containerID="1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7"
Sep 17 16:06:52.441443 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.441393    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7"} err="failed to get container status \"1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7\": rpc error: code = NotFound desc = could not find container \"1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7\": container with ID starting with 1a2c7d3a64a1e3a9bb22d511c0c4b7ff02541db03c9d615ce3d4013884c95da7 not found: ID does not exist"
Sep 17 16:06:52.516984 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.516953    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:52.516984 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.516979    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zl2d\" (UniqueName: \"kubernetes.io/projected/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-api-access-5zl2d\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:52.517166 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.516992    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:52.517166 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.517007    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c675c3c2-4894-4cbe-88c2-e8136f0090e8-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:52.517166 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.517018    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c675c3c2-4894-4cbe-88c2-e8136f0090e8-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:06:52.737753 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.737721    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"]
Sep 17 16:06:52.740780 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:52.740757    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj"]
Sep 17 16:06:53.309194 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:53.309148    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-g26kj" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": context deadline exceeded"
Sep 17 16:06:53.674860 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:53.674831    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" path="/var/lib/kubelet/pods/c675c3c2-4894-4cbe-88c2-e8136f0090e8/volumes"
Sep 17 16:06:56.041656 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.041625    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:56.075362 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.075331    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"]
Sep 17 16:06:56.075616 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.075604    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kube-rbac-proxy"
Sep 17 16:06:56.075664 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.075618    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kube-rbac-proxy"
Sep 17 16:06:56.075664 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.075628    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="storage-initializer"
Sep 17 16:06:56.075664 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.075635    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="storage-initializer"
Sep 17 16:06:56.075664 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.075645    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kserve-container"
Sep 17 16:06:56.075664 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.075651    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kserve-container"
Sep 17 16:06:56.075836 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.075708    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kube-rbac-proxy"
Sep 17 16:06:56.075836 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.075716    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c675c3c2-4894-4cbe-88c2-e8136f0090e8" containerName="kserve-container"
Sep 17 16:06:56.080303 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.080284    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.082406 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.082386    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:06:56.082497 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.082429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:06:56.082560 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.082516    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 16:06:56.082604 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.082559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:06:56.082985 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.082967    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 16:06:56.083043 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.082967    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:06:56.083214 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.083197    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:06:56.092791 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.092771    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"]
Sep 17 16:06:56.117068 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.117045    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"]
Sep 17 16:06:56.120156 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.120142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.121912 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.121894    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:56.123227 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.123212    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 16:06:56.131135 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.131116    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"]
Sep 17 16:06:56.145957 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.145936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24e000a-73af-4f14-9fdc-c2f270ff74b8-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.146046 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.145964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.146046 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.145994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.146123 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.146071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6xp\" (UniqueName: \"kubernetes.io/projected/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-api-access-wg6xp\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.146123 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.146104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.247237 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6xp\" (UniqueName: \"kubernetes.io/projected/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-api-access-wg6xp\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.247368 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c17377-a382-4dda-af4b-61a9dc9f957f-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.247368 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c17377-a382-4dda-af4b-61a9dc9f957f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.247368 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.247368 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24e000a-73af-4f14-9fdc-c2f270ff74b8-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.247542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.247542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.247542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8dh\" (UniqueName: \"kubernetes.io/projected/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-api-access-5z8dh\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.247542 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:06:56.247468    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 16:06:56.247542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.247731 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:06:56.247557    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24e000a-73af-4f14-9fdc-c2f270ff74b8-proxy-tls podName:d24e000a-73af-4f14-9fdc-c2f270ff74b8 nodeName:}" failed. No retries permitted until 2026-09-17 16:06:56.747536361 +0000 UTC m=+1701.589899145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d24e000a-73af-4f14-9fdc-c2f270ff74b8-proxy-tls") pod "isvc-canary-create-v2-predictor-b59dcb89c-4968k" (UID: "d24e000a-73af-4f14-9fdc-c2f270ff74b8") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 16:06:56.247731 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.247841 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.247971 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.247955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.248039 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.248024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.256537 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.256512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6xp\" (UniqueName: \"kubernetes.io/projected/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-api-access-wg6xp\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.348719 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.348644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.348719 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.348676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8dh\" (UniqueName: \"kubernetes.io/projected/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-api-access-5z8dh\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.348719 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.348707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.349014 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.348765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c17377-a382-4dda-af4b-61a9dc9f957f-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.349014 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.348814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c17377-a382-4dda-af4b-61a9dc9f957f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.349244 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.349217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c17377-a382-4dda-af4b-61a9dc9f957f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.349375 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.349356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.349614 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.349471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.351265 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.351244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c17377-a382-4dda-af4b-61a9dc9f957f-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.358917 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.358895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8dh\" (UniqueName: \"kubernetes.io/projected/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-api-access-5z8dh\") pod \"isvc-canary-create-predictor-5674b8c456-f6s8t\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.411505 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.411479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:56.429929 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.429906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:06:56.588982 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.588953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:56.704998 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.704892    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"]
Sep 17 16:06:56.707586 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:06:56.707562    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c17377_a382_4dda_af4b_61a9dc9f957f.slice/crio-2fa88a9524f38cd5529911444300325e72fc637f0a7bcf5d4598a476f7500d64 WatchSource:0}: Error finding container 2fa88a9524f38cd5529911444300325e72fc637f0a7bcf5d4598a476f7500d64: Status 404 returned error can't find the container with id 2fa88a9524f38cd5529911444300325e72fc637f0a7bcf5d4598a476f7500d64
Sep 17 16:06:56.751290 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.751266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24e000a-73af-4f14-9fdc-c2f270ff74b8-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.753507 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.753485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24e000a-73af-4f14-9fdc-c2f270ff74b8-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-4968k\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:56.990450 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:56.990342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:06:57.264548 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:57.264517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"]
Sep 17 16:06:57.267271 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:06:57.267245    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24e000a_73af_4f14_9fdc_c2f270ff74b8.slice/crio-8b6fd51ce9ebb13c437889918ce562b7026abafd80744af1d7f5019eaa80b279 WatchSource:0}: Error finding container 8b6fd51ce9ebb13c437889918ce562b7026abafd80744af1d7f5019eaa80b279: Status 404 returned error can't find the container with id 8b6fd51ce9ebb13c437889918ce562b7026abafd80744af1d7f5019eaa80b279
Sep 17 16:06:57.430513 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:57.430371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" event={"ID":"d24e000a-73af-4f14-9fdc-c2f270ff74b8","Type":"ContainerStarted","Data":"4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513"}
Sep 17 16:06:57.430699 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:57.430536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" event={"ID":"d24e000a-73af-4f14-9fdc-c2f270ff74b8","Type":"ContainerStarted","Data":"8b6fd51ce9ebb13c437889918ce562b7026abafd80744af1d7f5019eaa80b279"}
Sep 17 16:06:57.432196 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:57.432162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" event={"ID":"45c17377-a382-4dda-af4b-61a9dc9f957f","Type":"ContainerStarted","Data":"b76911ab2f0bdc9693982648ab28c9f08981b321714aced03aed71b388015394"}
Sep 17 16:06:57.432325 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:06:57.432215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" event={"ID":"45c17377-a382-4dda-af4b-61a9dc9f957f","Type":"ContainerStarted","Data":"2fa88a9524f38cd5529911444300325e72fc637f0a7bcf5d4598a476f7500d64"}
Sep 17 16:07:01.450737 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:01.450702    2580 generic.go:356] "Generic (PLEG): container finished" podID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerID="b76911ab2f0bdc9693982648ab28c9f08981b321714aced03aed71b388015394" exitCode=0
Sep 17 16:07:01.451234 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:01.450757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" event={"ID":"45c17377-a382-4dda-af4b-61a9dc9f957f","Type":"ContainerDied","Data":"b76911ab2f0bdc9693982648ab28c9f08981b321714aced03aed71b388015394"}
Sep 17 16:07:01.451954 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:01.451911    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:07:02.462854 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:02.462276    2580 generic.go:356] "Generic (PLEG): container finished" podID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerID="4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513" exitCode=0
Sep 17 16:07:02.462854 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:02.462391    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" event={"ID":"d24e000a-73af-4f14-9fdc-c2f270ff74b8","Type":"ContainerDied","Data":"4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513"}
Sep 17 16:07:10.635738 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:10.635706    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:15.818075 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:15.818038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:15.985438 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:15.985402    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:17.554195 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.554154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" event={"ID":"d24e000a-73af-4f14-9fdc-c2f270ff74b8","Type":"ContainerStarted","Data":"9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1"}
Sep 17 16:07:17.554626 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.554269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" event={"ID":"d24e000a-73af-4f14-9fdc-c2f270ff74b8","Type":"ContainerStarted","Data":"689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2"}
Sep 17 16:07:17.554626 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.554607    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:07:17.554746 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.554665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:07:17.556326 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.556297    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:07:17.556492 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.556470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" event={"ID":"45c17377-a382-4dda-af4b-61a9dc9f957f","Type":"ContainerStarted","Data":"1c8f41ac2ebcc560db1d2b50c0d70429796bc9389c30750fb1477506b1a74f6f"}
Sep 17 16:07:17.556542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.556519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" event={"ID":"45c17377-a382-4dda-af4b-61a9dc9f957f","Type":"ContainerStarted","Data":"00107d033b7430753552cb89b94cbc31e544160066e81dca5b646a4155cf3c54"}
Sep 17 16:07:17.556773 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.556759    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:07:17.574959 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.574917    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podStartSLOduration=2.708601088 podStartE2EDuration="21.574902486s" totalImagesPullingTime="14.721506727s" totalInitContainerRuntime="4.144794671s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:17.574526853 +0000 UTC m=+1722.416889659" watchObservedRunningTime="2026-09-17 16:07:17.574902486 +0000 UTC m=+1722.417265288"
Sep 17 16:07:17.593742 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:17.593702    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podStartSLOduration=1.95847617 podStartE2EDuration="21.593686536s" totalImagesPullingTime="15.712335395s" totalInitContainerRuntime="3.922874971s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:17.593151752 +0000 UTC m=+1722.435514556" watchObservedRunningTime="2026-09-17 16:07:17.593686536 +0000 UTC m=+1722.436049324"
Sep 17 16:07:18.561064 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:18.561021    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:07:18.561546 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:18.561332    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:07:18.562517 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:18.562491    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:07:19.565792 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:19.565749    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:07:23.565835 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:23.565782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:07:23.566400 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:23.566365    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:07:24.570656 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:24.570627    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:07:24.571187 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:24.571160    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:07:33.566413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:33.566370    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:07:34.571469 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:34.571417    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:07:43.566514 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:43.566471    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:07:44.571442 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:44.571387    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:07:53.566674 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:53.566635    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:07:54.571228 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:07:54.571182    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:08:03.566422 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:03.566381    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:08:04.571170 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:04.571131    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:08:13.566612 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:13.566560    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:08:14.571992 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:14.571946    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:08:15.921535 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:15.921499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:21.110851 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:21.110789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:21.558221 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:21.558134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:23.566238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:23.566200    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:08:24.571454 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:24.571401    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:08:24.674347 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:24.674298    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:08:34.571817 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:34.571778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:08:34.675223 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:34.675196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:08:34.836259 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:34.836183    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"]
Sep 17 16:08:34.853293 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:34.853240    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container" containerID="cri-o://00107d033b7430753552cb89b94cbc31e544160066e81dca5b646a4155cf3c54" gracePeriod=30
Sep 17 16:08:34.853454 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:34.853267    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kube-rbac-proxy" containerID="cri-o://1c8f41ac2ebcc560db1d2b50c0d70429796bc9389c30750fb1477506b1a74f6f" gracePeriod=30
Sep 17 16:08:35.706024 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:35.705990    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:08:35.706613 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:35.706596    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:08:35.859762 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:35.859732    2580 generic.go:356] "Generic (PLEG): container finished" podID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerID="1c8f41ac2ebcc560db1d2b50c0d70429796bc9389c30750fb1477506b1a74f6f" exitCode=2
Sep 17 16:08:35.859932 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:35.859788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" event={"ID":"45c17377-a382-4dda-af4b-61a9dc9f957f","Type":"ContainerDied","Data":"1c8f41ac2ebcc560db1d2b50c0d70429796bc9389c30750fb1477506b1a74f6f"}
Sep 17 16:08:36.296360 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:36.296323    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"]
Sep 17 16:08:36.296636 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:36.296594    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container" containerID="cri-o://689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2" gracePeriod=30
Sep 17 16:08:36.296729 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:36.296672    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kube-rbac-proxy" containerID="cri-o://9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1" gracePeriod=30
Sep 17 16:08:36.311788 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:36.311762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:36.866384 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:36.866348    2580 generic.go:356] "Generic (PLEG): container finished" podID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerID="9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1" exitCode=2
Sep 17 16:08:36.866771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:36.866390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" event={"ID":"d24e000a-73af-4f14-9fdc-c2f270ff74b8","Type":"ContainerDied","Data":"9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1"}
Sep 17 16:08:38.561635 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:38.561592    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 17 16:08:39.566385 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:39.566334    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 16:08:39.883288 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:39.883255    2580 generic.go:356] "Generic (PLEG): container finished" podID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerID="00107d033b7430753552cb89b94cbc31e544160066e81dca5b646a4155cf3c54" exitCode=0
Sep 17 16:08:39.883465 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:39.883318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" event={"ID":"45c17377-a382-4dda-af4b-61a9dc9f957f","Type":"ContainerDied","Data":"00107d033b7430753552cb89b94cbc31e544160066e81dca5b646a4155cf3c54"}
Sep 17 16:08:39.990996 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:39.990970    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:08:40.112404 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.112364    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-cabundle-cert\") pod \"45c17377-a382-4dda-af4b-61a9dc9f957f\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") "
Sep 17 16:08:40.112590 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.112455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-rbac-proxy-sar-config\") pod \"45c17377-a382-4dda-af4b-61a9dc9f957f\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") "
Sep 17 16:08:40.112590 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.112474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z8dh\" (UniqueName: \"kubernetes.io/projected/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-api-access-5z8dh\") pod \"45c17377-a382-4dda-af4b-61a9dc9f957f\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") "
Sep 17 16:08:40.112590 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.112495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c17377-a382-4dda-af4b-61a9dc9f957f-kserve-provision-location\") pod \"45c17377-a382-4dda-af4b-61a9dc9f957f\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") "
Sep 17 16:08:40.112590 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.112520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c17377-a382-4dda-af4b-61a9dc9f957f-proxy-tls\") pod \"45c17377-a382-4dda-af4b-61a9dc9f957f\" (UID: \"45c17377-a382-4dda-af4b-61a9dc9f957f\") "
Sep 17 16:08:40.112881 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.112852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45c17377-a382-4dda-af4b-61a9dc9f957f" (UID: "45c17377-a382-4dda-af4b-61a9dc9f957f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:40.112881 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.112859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c17377-a382-4dda-af4b-61a9dc9f957f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45c17377-a382-4dda-af4b-61a9dc9f957f" (UID: "45c17377-a382-4dda-af4b-61a9dc9f957f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:40.112997 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.112910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45c17377-a382-4dda-af4b-61a9dc9f957f" (UID: "45c17377-a382-4dda-af4b-61a9dc9f957f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:40.114912 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.114847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-api-access-5z8dh" (OuterVolumeSpecName: "kube-api-access-5z8dh") pod "45c17377-a382-4dda-af4b-61a9dc9f957f" (UID: "45c17377-a382-4dda-af4b-61a9dc9f957f"). InnerVolumeSpecName "kube-api-access-5z8dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:40.114912 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.114869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c17377-a382-4dda-af4b-61a9dc9f957f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45c17377-a382-4dda-af4b-61a9dc9f957f" (UID: "45c17377-a382-4dda-af4b-61a9dc9f957f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:40.213764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.213721    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:40.213764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.213754    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z8dh\" (UniqueName: \"kubernetes.io/projected/45c17377-a382-4dda-af4b-61a9dc9f957f-kube-api-access-5z8dh\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:40.213764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.213763    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45c17377-a382-4dda-af4b-61a9dc9f957f-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:40.213764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.213775    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c17377-a382-4dda-af4b-61a9dc9f957f-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:40.214068 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.213783    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45c17377-a382-4dda-af4b-61a9dc9f957f-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:40.889682 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.889642    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t" event={"ID":"45c17377-a382-4dda-af4b-61a9dc9f957f","Type":"ContainerDied","Data":"2fa88a9524f38cd5529911444300325e72fc637f0a7bcf5d4598a476f7500d64"}
Sep 17 16:08:40.890140 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.889717    2580 scope.go:117] "RemoveContainer" containerID="1c8f41ac2ebcc560db1d2b50c0d70429796bc9389c30750fb1477506b1a74f6f"
Sep 17 16:08:40.890140 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.889654    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"
Sep 17 16:08:40.898479 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.898460    2580 scope.go:117] "RemoveContainer" containerID="00107d033b7430753552cb89b94cbc31e544160066e81dca5b646a4155cf3c54"
Sep 17 16:08:40.905735 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.905714    2580 scope.go:117] "RemoveContainer" containerID="b76911ab2f0bdc9693982648ab28c9f08981b321714aced03aed71b388015394"
Sep 17 16:08:40.917136 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.917112    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"]
Sep 17 16:08:40.924210 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:40.924191    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-f6s8t"]
Sep 17 16:08:41.480771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.480743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:41.553515 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.553481    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"]
Sep 17 16:08:41.553779 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.553768    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container"
Sep 17 16:08:41.553851 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.553781    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container"
Sep 17 16:08:41.553851 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.553817    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kube-rbac-proxy"
Sep 17 16:08:41.553851 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.553823    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kube-rbac-proxy"
Sep 17 16:08:41.553851 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.553831    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="storage-initializer"
Sep 17 16:08:41.553851 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.553837    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="storage-initializer"
Sep 17 16:08:41.554007 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.553889    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kube-rbac-proxy"
Sep 17 16:08:41.554007 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.553898    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" containerName="kserve-container"
Sep 17 16:08:41.557668 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.557649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.562595 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.562573    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 16:08:41.562965 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.562945    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 16:08:41.579199 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.579179    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"]
Sep 17 16:08:41.609461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.609434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:41.625547 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.625523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/984d4a56-7f43-4d6a-80b6-b81a33b880fd-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.625665 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.625552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmnbj\" (UniqueName: \"kubernetes.io/projected/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-api-access-xmnbj\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.625665 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.625610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.625665 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.625630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.625886 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.625755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.674662 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.674633    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c17377-a382-4dda-af4b-61a9dc9f957f" path="/var/lib/kubelet/pods/45c17377-a382-4dda-af4b-61a9dc9f957f/volumes"
Sep 17 16:08:41.727229 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.727200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.727369 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.727234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.727369 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.727279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.727369 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.727311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/984d4a56-7f43-4d6a-80b6-b81a33b880fd-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.727369 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.727328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmnbj\" (UniqueName: \"kubernetes.io/projected/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-api-access-xmnbj\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.727663 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.727641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.727951 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.727925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.728068 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.728047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.730150 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.730126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/984d4a56-7f43-4d6a-80b6-b81a33b880fd-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.749362 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.749340    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"]
Sep 17 16:08:41.751200 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.751185    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:08:41.752558 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.752544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.754632 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.754609    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 16:08:41.755562 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.755544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmnbj\" (UniqueName: \"kubernetes.io/projected/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-api-access-xmnbj\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-vws2k\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.775545 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.775522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:41.781868 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.781846    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"]
Sep 17 16:08:41.827926 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.827890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-rbac-proxy-sar-config\") pod \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") "
Sep 17 16:08:41.828100 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.827953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kserve-provision-location\") pod \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") "
Sep 17 16:08:41.828100 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828006    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-cabundle-cert\") pod \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") "
Sep 17 16:08:41.828100 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828034    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24e000a-73af-4f14-9fdc-c2f270ff74b8-proxy-tls\") pod \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") "
Sep 17 16:08:41.828100 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg6xp\" (UniqueName: \"kubernetes.io/projected/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-api-access-wg6xp\") pod \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\" (UID: \"d24e000a-73af-4f14-9fdc-c2f270ff74b8\") "
Sep 17 16:08:41.828304 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdnlv\" (UniqueName: \"kubernetes.io/projected/b3fca2d3-a468-4b16-8218-e6560650b719-kube-api-access-mdnlv\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.828304 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.828413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828314    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d24e000a-73af-4f14-9fdc-c2f270ff74b8" (UID: "d24e000a-73af-4f14-9fdc-c2f270ff74b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:41.828413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3fca2d3-a468-4b16-8218-e6560650b719-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.828413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828349    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d24e000a-73af-4f14-9fdc-c2f270ff74b8" (UID: "d24e000a-73af-4f14-9fdc-c2f270ff74b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:41.828413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.828585 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d24e000a-73af-4f14-9fdc-c2f270ff74b8" (UID: "d24e000a-73af-4f14-9fdc-c2f270ff74b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:41.828585 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3fca2d3-a468-4b16-8218-e6560650b719-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.828585 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828498    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.828585 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828514    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.828585 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.828527    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.830272 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.830245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24e000a-73af-4f14-9fdc-c2f270ff74b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d24e000a-73af-4f14-9fdc-c2f270ff74b8" (UID: "d24e000a-73af-4f14-9fdc-c2f270ff74b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:41.830374 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.830303    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-api-access-wg6xp" (OuterVolumeSpecName: "kube-api-access-wg6xp") pod "d24e000a-73af-4f14-9fdc-c2f270ff74b8" (UID: "d24e000a-73af-4f14-9fdc-c2f270ff74b8"). InnerVolumeSpecName "kube-api-access-wg6xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:41.868494 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.868467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:41.891457 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.891420    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:41.897316 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.897295    2580 generic.go:356] "Generic (PLEG): container finished" podID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerID="689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2" exitCode=0
Sep 17 16:08:41.897414 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.897352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" event={"ID":"d24e000a-73af-4f14-9fdc-c2f270ff74b8","Type":"ContainerDied","Data":"689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2"}
Sep 17 16:08:41.897414 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.897376    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"
Sep 17 16:08:41.897541 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.897427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k" event={"ID":"d24e000a-73af-4f14-9fdc-c2f270ff74b8","Type":"ContainerDied","Data":"8b6fd51ce9ebb13c437889918ce562b7026abafd80744af1d7f5019eaa80b279"}
Sep 17 16:08:41.897541 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.897454    2580 scope.go:117] "RemoveContainer" containerID="9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1"
Sep 17 16:08:41.906366 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.906347    2580 scope.go:117] "RemoveContainer" containerID="689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2"
Sep 17 16:08:41.917079 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.917038    2580 scope.go:117] "RemoveContainer" containerID="4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513"
Sep 17 16:08:41.929729 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.928896    2580 scope.go:117] "RemoveContainer" containerID="9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1"
Sep 17 16:08:41.929729 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.929145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.929729 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.929190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3fca2d3-a468-4b16-8218-e6560650b719-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.929729 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.929217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.929729 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.929249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3fca2d3-a468-4b16-8218-e6560650b719-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.929729 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.929301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdnlv\" (UniqueName: \"kubernetes.io/projected/b3fca2d3-a468-4b16-8218-e6560650b719-kube-api-access-mdnlv\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.929729 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.929383    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg6xp\" (UniqueName: \"kubernetes.io/projected/d24e000a-73af-4f14-9fdc-c2f270ff74b8-kube-api-access-wg6xp\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.929729 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.929402    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24e000a-73af-4f14-9fdc-c2f270ff74b8-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.930479 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.930404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.930689 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.930644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3fca2d3-a468-4b16-8218-e6560650b719-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.930943 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:08:41.930902    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1\": container with ID starting with 9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1 not found: ID does not exist" containerID="9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1"
Sep 17 16:08:41.931025 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.930945    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1"} err="failed to get container status \"9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1\": rpc error: code = NotFound desc = could not find container \"9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1\": container with ID starting with 9e8e87b2ff034e9bd1ef8b02c3d734b73fb1ed9ae7a3f05d4455d95dda6a48c1 not found: ID does not exist"
Sep 17 16:08:41.931025 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.930972    2580 scope.go:117] "RemoveContainer" containerID="689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2"
Sep 17 16:08:41.931231 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.931201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.931330 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:08:41.931302    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2\": container with ID starting with 689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2 not found: ID does not exist" containerID="689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2"
Sep 17 16:08:41.931407 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.931338    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2"} err="failed to get container status \"689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2\": rpc error: code = NotFound desc = could not find container \"689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2\": container with ID starting with 689d2758632c3ffde55b5a12b3fa12616a70b7f98e4ae6f5b9050d3ec7dcecb2 not found: ID does not exist"
Sep 17 16:08:41.931407 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.931360    2580 scope.go:117] "RemoveContainer" containerID="4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513"
Sep 17 16:08:41.931665 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:08:41.931637    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513\": container with ID starting with 4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513 not found: ID does not exist" containerID="4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513"
Sep 17 16:08:41.931757 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.931673    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513"} err="failed to get container status \"4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513\": rpc error: code = NotFound desc = could not find container \"4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513\": container with ID starting with 4a89be9c4ee72423210555b2d5e8b01a6464ddc13100a1aabb81b7211b49b513 not found: ID does not exist"
Sep 17 16:08:41.933218 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.933195    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"]
Sep 17 16:08:41.934317 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.934290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3fca2d3-a468-4b16-8218-e6560650b719-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.942412 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.942380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdnlv\" (UniqueName: \"kubernetes.io/projected/b3fca2d3-a468-4b16-8218-e6560650b719-kube-api-access-mdnlv\") pod \"isvc-canary-promote-predictor-8499c568cb-s5bz7\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:41.947885 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:41.947865    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-4968k"]
Sep 17 16:08:42.067951 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:42.067854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:42.150847 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:42.150818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"]
Sep 17 16:08:42.152280 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:08:42.152260    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984d4a56_7f43_4d6a_80b6_b81a33b880fd.slice/crio-1b56bfd60c9abc0bbde79862b79c127006abe80df853ff79cd40d3ffe0529440 WatchSource:0}: Error finding container 1b56bfd60c9abc0bbde79862b79c127006abe80df853ff79cd40d3ffe0529440: Status 404 returned error can't find the container with id 1b56bfd60c9abc0bbde79862b79c127006abe80df853ff79cd40d3ffe0529440
Sep 17 16:08:42.346485 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:42.346400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"]
Sep 17 16:08:42.348532 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:08:42.348505    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fca2d3_a468_4b16_8218_e6560650b719.slice/crio-b0d95dd0d28f15703a26844f10f3e86bb77f43d5d0a501a7617ba118fde17415 WatchSource:0}: Error finding container b0d95dd0d28f15703a26844f10f3e86bb77f43d5d0a501a7617ba118fde17415: Status 404 returned error can't find the container with id b0d95dd0d28f15703a26844f10f3e86bb77f43d5d0a501a7617ba118fde17415
Sep 17 16:08:42.904017 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:42.903977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" event={"ID":"984d4a56-7f43-4d6a-80b6-b81a33b880fd","Type":"ContainerStarted","Data":"9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f"}
Sep 17 16:08:42.904491 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:42.904061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" event={"ID":"984d4a56-7f43-4d6a-80b6-b81a33b880fd","Type":"ContainerStarted","Data":"1b56bfd60c9abc0bbde79862b79c127006abe80df853ff79cd40d3ffe0529440"}
Sep 17 16:08:42.906749 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:42.906714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" event={"ID":"b3fca2d3-a468-4b16-8218-e6560650b719","Type":"ContainerStarted","Data":"0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0"}
Sep 17 16:08:42.906918 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:42.906790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" event={"ID":"b3fca2d3-a468-4b16-8218-e6560650b719","Type":"ContainerStarted","Data":"b0d95dd0d28f15703a26844f10f3e86bb77f43d5d0a501a7617ba118fde17415"}
Sep 17 16:08:43.675029 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:43.674994    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" path="/var/lib/kubelet/pods/d24e000a-73af-4f14-9fdc-c2f270ff74b8/volumes"
Sep 17 16:08:46.924761 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:46.924720    2580 generic.go:356] "Generic (PLEG): container finished" podID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerID="9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f" exitCode=0
Sep 17 16:08:46.925237 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:46.924791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" event={"ID":"984d4a56-7f43-4d6a-80b6-b81a33b880fd","Type":"ContainerDied","Data":"9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f"}
Sep 17 16:08:46.926463 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:46.926440    2580 generic.go:356] "Generic (PLEG): container finished" podID="b3fca2d3-a468-4b16-8218-e6560650b719" containerID="0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0" exitCode=0
Sep 17 16:08:46.926545 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:46.926506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" event={"ID":"b3fca2d3-a468-4b16-8218-e6560650b719","Type":"ContainerDied","Data":"0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0"}
Sep 17 16:08:47.933270 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.933229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" event={"ID":"984d4a56-7f43-4d6a-80b6-b81a33b880fd","Type":"ContainerStarted","Data":"55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a"}
Sep 17 16:08:47.933751 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.933305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" event={"ID":"984d4a56-7f43-4d6a-80b6-b81a33b880fd","Type":"ContainerStarted","Data":"893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c"}
Sep 17 16:08:47.933751 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.933602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:47.935424 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.935393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" event={"ID":"b3fca2d3-a468-4b16-8218-e6560650b719","Type":"ContainerStarted","Data":"1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5"}
Sep 17 16:08:47.935562 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.935454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" event={"ID":"b3fca2d3-a468-4b16-8218-e6560650b719","Type":"ContainerStarted","Data":"56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a"}
Sep 17 16:08:47.935743 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.935722    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:47.935829 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.935775    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:47.937094 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.937067    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:08:47.962347 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.962300    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podStartSLOduration=3.170286547 podStartE2EDuration="6.962288085s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.792001538s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:08:47.960375803 +0000 UTC m=+1812.802738608" watchObservedRunningTime="2026-09-17 16:08:47.962288085 +0000 UTC m=+1812.804650888"
Sep 17 16:08:47.987661 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:47.987618    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podStartSLOduration=3.052377654 podStartE2EDuration="6.987604107s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.935226453s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:08:47.986270676 +0000 UTC m=+1812.828633481" watchObservedRunningTime="2026-09-17 16:08:47.987604107 +0000 UTC m=+1812.829966912"
Sep 17 16:08:48.940440 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:48.940390    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:08:48.940903 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:48.940633    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:48.941855 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:48.941830    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:08:49.944488 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:49.944446    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:08:53.945056 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:53.945027    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:08:53.945562 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:53.945535    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:08:54.951057 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:54.951028    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:08:54.951514 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:08:54.951490    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:09:03.945883 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:03.945848    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:09:04.951941 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:04.951900    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:09:13.946468 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:13.946422    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:09:14.952365 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:14.952326    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:09:23.945719 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:23.945677    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:09:24.952390 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:24.952345    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:09:33.946123 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:33.946084    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:09:34.951576 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:34.951533    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:09:42.395549 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:42.395520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:43.945524 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:43.945480    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:09:44.952063 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:44.952022    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:09:51.321091 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:51.321058    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:53.946033 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:53.945990    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:09:54.951957 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:54.951917    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:09:56.437131 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.437093    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:56.478917 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.478888    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"]
Sep 17 16:09:56.479193 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.479182    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kube-rbac-proxy"
Sep 17 16:09:56.479236 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.479195    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kube-rbac-proxy"
Sep 17 16:09:56.479236 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.479204    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container"
Sep 17 16:09:56.479236 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.479210    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container"
Sep 17 16:09:56.479236 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.479223    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="storage-initializer"
Sep 17 16:09:56.479236 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.479228    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="storage-initializer"
Sep 17 16:09:56.479442 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.479285    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kserve-container"
Sep 17 16:09:56.479442 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.479293    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24e000a-73af-4f14-9fdc-c2f270ff74b8" containerName="kube-rbac-proxy"
Sep 17 16:09:56.482312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.482294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.484876 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.484856    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 16:09:56.485234 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.485219    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 16:09:56.497169 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.497142    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"]
Sep 17 16:09:56.574084 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.574057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:56.582722 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.582701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q498p\" (UniqueName: \"kubernetes.io/projected/04c31303-4709-4f8b-86e3-c2288254f54d-kube-api-access-q498p\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.582857 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.582729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.582857 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.582755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04c31303-4709-4f8b-86e3-c2288254f54d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.582945 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.582866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04c31303-4709-4f8b-86e3-c2288254f54d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.582945 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.582920    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.683466 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.683435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q498p\" (UniqueName: \"kubernetes.io/projected/04c31303-4709-4f8b-86e3-c2288254f54d-kube-api-access-q498p\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.683466 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.683467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.683691 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.683486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04c31303-4709-4f8b-86e3-c2288254f54d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.683691 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.683515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04c31303-4709-4f8b-86e3-c2288254f54d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.683691 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.683559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.684096 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.684065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04c31303-4709-4f8b-86e3-c2288254f54d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.684254 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.684234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.684302 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.684244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.686110 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.686088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04c31303-4709-4f8b-86e3-c2288254f54d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.694431 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.694358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q498p\" (UniqueName: \"kubernetes.io/projected/04c31303-4709-4f8b-86e3-c2288254f54d-kube-api-access-q498p\") pod \"isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:56.792065 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:56.792025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:09:57.075120 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:57.075060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"]
Sep 17 16:09:57.077818 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:09:57.077765    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c31303_4709_4f8b_86e3_c2288254f54d.slice/crio-788b77319e52b3191955696be22dc69679320f4df82718aecb4a923259ce6ae2 WatchSource:0}: Error finding container 788b77319e52b3191955696be22dc69679320f4df82718aecb4a923259ce6ae2: Status 404 returned error can't find the container with id 788b77319e52b3191955696be22dc69679320f4df82718aecb4a923259ce6ae2
Sep 17 16:09:57.218654 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:57.218616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" event={"ID":"04c31303-4709-4f8b-86e3-c2288254f54d","Type":"ContainerStarted","Data":"3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637"}
Sep 17 16:09:57.218853 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:09:57.218685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" event={"ID":"04c31303-4709-4f8b-86e3-c2288254f54d","Type":"ContainerStarted","Data":"788b77319e52b3191955696be22dc69679320f4df82718aecb4a923259ce6ae2"}
Sep 17 16:10:01.235572 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:01.235536    2580 generic.go:356] "Generic (PLEG): container finished" podID="04c31303-4709-4f8b-86e3-c2288254f54d" containerID="3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637" exitCode=0
Sep 17 16:10:01.235964 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:01.235608    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" event={"ID":"04c31303-4709-4f8b-86e3-c2288254f54d","Type":"ContainerDied","Data":"3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637"}
Sep 17 16:10:02.241865 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:02.241832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" event={"ID":"04c31303-4709-4f8b-86e3-c2288254f54d","Type":"ContainerStarted","Data":"3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005"}
Sep 17 16:10:02.242272 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:02.241898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" event={"ID":"04c31303-4709-4f8b-86e3-c2288254f54d","Type":"ContainerStarted","Data":"29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404"}
Sep 17 16:10:02.242272 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:02.242102    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:10:02.266764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:02.266701    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podStartSLOduration=2.278887334 podStartE2EDuration="6.266678406s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.987791072s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:02.265233025 +0000 UTC m=+1887.107595831" watchObservedRunningTime="2026-09-17 16:10:02.266678406 +0000 UTC m=+1887.109041213"
Sep 17 16:10:03.246313 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:03.246281    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:10:03.247831 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:03.247787    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:10:03.675535 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:03.675507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:10:04.250408 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:04.250373    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:10:04.670468 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:04.670442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:10:09.256227 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:09.256194    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:10:09.256661 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:09.256637    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:10:11.518008 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:11.517976    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"]
Sep 17 16:10:11.518388 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:11.518249    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" containerID="cri-o://56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a" gracePeriod=30
Sep 17 16:10:11.518388 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:11.518286    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kube-rbac-proxy" containerID="cri-o://1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5" gracePeriod=30
Sep 17 16:10:11.549931 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:11.549900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:12.284598 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:12.284562    2580 generic.go:356] "Generic (PLEG): container finished" podID="b3fca2d3-a468-4b16-8218-e6560650b719" containerID="1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5" exitCode=2
Sep 17 16:10:12.284765 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:12.284619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" event={"ID":"b3fca2d3-a468-4b16-8218-e6560650b719","Type":"ContainerDied","Data":"1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5"}
Sep 17 16:10:13.670817 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:13.670763    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 16:10:13.941400 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:13.941310    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 16:10:16.564618 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.564587    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:10:16.621438 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.621357    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdnlv\" (UniqueName: \"kubernetes.io/projected/b3fca2d3-a468-4b16-8218-e6560650b719-kube-api-access-mdnlv\") pod \"b3fca2d3-a468-4b16-8218-e6560650b719\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") "
Sep 17 16:10:16.621438 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.621391    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-cabundle-cert\") pod \"b3fca2d3-a468-4b16-8218-e6560650b719\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") "
Sep 17 16:10:16.621438 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.621412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3fca2d3-a468-4b16-8218-e6560650b719-proxy-tls\") pod \"b3fca2d3-a468-4b16-8218-e6560650b719\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") "
Sep 17 16:10:16.621438 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.621428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-kube-rbac-proxy-sar-config\") pod \"b3fca2d3-a468-4b16-8218-e6560650b719\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") "
Sep 17 16:10:16.621713 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.621616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3fca2d3-a468-4b16-8218-e6560650b719-kserve-provision-location\") pod \"b3fca2d3-a468-4b16-8218-e6560650b719\" (UID: \"b3fca2d3-a468-4b16-8218-e6560650b719\") "
Sep 17 16:10:16.621872 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.621839    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3fca2d3-a468-4b16-8218-e6560650b719" (UID: "b3fca2d3-a468-4b16-8218-e6560650b719"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:16.622354 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.621906    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3fca2d3-a468-4b16-8218-e6560650b719" (UID: "b3fca2d3-a468-4b16-8218-e6560650b719"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:16.622354 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.622252    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3fca2d3-a468-4b16-8218-e6560650b719-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3fca2d3-a468-4b16-8218-e6560650b719" (UID: "b3fca2d3-a468-4b16-8218-e6560650b719"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:16.627657 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.624220    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3fca2d3-a468-4b16-8218-e6560650b719-kube-api-access-mdnlv" (OuterVolumeSpecName: "kube-api-access-mdnlv") pod "b3fca2d3-a468-4b16-8218-e6560650b719" (UID: "b3fca2d3-a468-4b16-8218-e6560650b719"). InnerVolumeSpecName "kube-api-access-mdnlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:10:16.627769 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.627731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3fca2d3-a468-4b16-8218-e6560650b719-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3fca2d3-a468-4b16-8218-e6560650b719" (UID: "b3fca2d3-a468-4b16-8218-e6560650b719"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:10:16.722529 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.722491    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdnlv\" (UniqueName: \"kubernetes.io/projected/b3fca2d3-a468-4b16-8218-e6560650b719-kube-api-access-mdnlv\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:10:16.722529 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.722522    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:10:16.722529 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.722534    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3fca2d3-a468-4b16-8218-e6560650b719-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:10:16.722759 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.722543    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3fca2d3-a468-4b16-8218-e6560650b719-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:10:16.722759 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:16.722552    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3fca2d3-a468-4b16-8218-e6560650b719-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:10:17.306816 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.306774    2580 generic.go:356] "Generic (PLEG): container finished" podID="b3fca2d3-a468-4b16-8218-e6560650b719" containerID="56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a" exitCode=0
Sep 17 16:10:17.307010 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.306844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" event={"ID":"b3fca2d3-a468-4b16-8218-e6560650b719","Type":"ContainerDied","Data":"56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a"}
Sep 17 16:10:17.307010 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.306879    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"
Sep 17 16:10:17.307010 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.306908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7" event={"ID":"b3fca2d3-a468-4b16-8218-e6560650b719","Type":"ContainerDied","Data":"b0d95dd0d28f15703a26844f10f3e86bb77f43d5d0a501a7617ba118fde17415"}
Sep 17 16:10:17.307010 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.306934    2580 scope.go:117] "RemoveContainer" containerID="1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5"
Sep 17 16:10:17.315949 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.315934    2580 scope.go:117] "RemoveContainer" containerID="56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a"
Sep 17 16:10:17.325510 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.325488    2580 scope.go:117] "RemoveContainer" containerID="0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0"
Sep 17 16:10:17.329870 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.329846    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"]
Sep 17 16:10:17.333289 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.333271    2580 scope.go:117] "RemoveContainer" containerID="1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5"
Sep 17 16:10:17.333563 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:10:17.333546    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5\": container with ID starting with 1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5 not found: ID does not exist" containerID="1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5"
Sep 17 16:10:17.333636 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.333574    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5"} err="failed to get container status \"1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5\": rpc error: code = NotFound desc = could not find container \"1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5\": container with ID starting with 1e59ea92b49b835473a6ae20e66885514b7aa88ed54b6c88017261f6b2db8af5 not found: ID does not exist"
Sep 17 16:10:17.333636 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.333596    2580 scope.go:117] "RemoveContainer" containerID="56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a"
Sep 17 16:10:17.333886 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:10:17.333865    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a\": container with ID starting with 56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a not found: ID does not exist" containerID="56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a"
Sep 17 16:10:17.333934 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.333894    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a"} err="failed to get container status \"56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a\": rpc error: code = NotFound desc = could not find container \"56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a\": container with ID starting with 56af0c9575e856c5946c4e9d063a5fe7fd9827d20b6ff8bb7bd893535f64579a not found: ID does not exist"
Sep 17 16:10:17.333934 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.333908    2580 scope.go:117] "RemoveContainer" containerID="0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0"
Sep 17 16:10:17.334143 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:10:17.334125    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0\": container with ID starting with 0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0 not found: ID does not exist" containerID="0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0"
Sep 17 16:10:17.334187 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.334148    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0"} err="failed to get container status \"0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0\": rpc error: code = NotFound desc = could not find container \"0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0\": container with ID starting with 0a384d6b4172ad5f01d6f4c9bceb9e6c1296c4be35bcbf8922ecf9502f5c39a0 not found: ID does not exist"
Sep 17 16:10:17.338720 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.338698    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-s5bz7"]
Sep 17 16:10:17.674540 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:17.674499    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" path="/var/lib/kubelet/pods/b3fca2d3-a468-4b16-8218-e6560650b719/volumes"
Sep 17 16:10:19.257390 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:19.257353    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:10:29.257110 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:29.257072    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:10:39.256784 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:39.256740    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:10:49.257375 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:49.257333    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:10:59.257587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:10:59.257543    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:11:09.257563 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:09.257524    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:11:19.258027 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:19.257997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:11:26.654163 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:26.654135    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"]
Sep 17 16:11:26.654603 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:26.654357    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" containerID="cri-o://29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404" gracePeriod=30
Sep 17 16:11:26.654603 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:26.654380    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kube-rbac-proxy" containerID="cri-o://3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005" gracePeriod=30
Sep 17 16:11:26.672058 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:26.672029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:27.583641 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:27.583605    2580 generic.go:356] "Generic (PLEG): container finished" podID="04c31303-4709-4f8b-86e3-c2288254f54d" containerID="3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005" exitCode=2
Sep 17 16:11:27.583827 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:27.583681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" event={"ID":"04c31303-4709-4f8b-86e3-c2288254f54d","Type":"ContainerDied","Data":"3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005"}
Sep 17 16:11:29.250647 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:29.250608    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 16:11:29.257113 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:29.257082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:11:31.301309 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.301283    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:11:31.414970 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.414938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-cabundle-cert\") pod \"04c31303-4709-4f8b-86e3-c2288254f54d\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") "
Sep 17 16:11:31.415128 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.414983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04c31303-4709-4f8b-86e3-c2288254f54d-proxy-tls\") pod \"04c31303-4709-4f8b-86e3-c2288254f54d\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") "
Sep 17 16:11:31.415128 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.415028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q498p\" (UniqueName: \"kubernetes.io/projected/04c31303-4709-4f8b-86e3-c2288254f54d-kube-api-access-q498p\") pod \"04c31303-4709-4f8b-86e3-c2288254f54d\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") "
Sep 17 16:11:31.415128 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.415068    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-kube-rbac-proxy-sar-config\") pod \"04c31303-4709-4f8b-86e3-c2288254f54d\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") "
Sep 17 16:11:31.415128 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.415113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04c31303-4709-4f8b-86e3-c2288254f54d-kserve-provision-location\") pod \"04c31303-4709-4f8b-86e3-c2288254f54d\" (UID: \"04c31303-4709-4f8b-86e3-c2288254f54d\") "
Sep 17 16:11:31.415467 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.415444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04c31303-4709-4f8b-86e3-c2288254f54d" (UID: "04c31303-4709-4f8b-86e3-c2288254f54d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:31.415543 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.415499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04c31303-4709-4f8b-86e3-c2288254f54d" (UID: "04c31303-4709-4f8b-86e3-c2288254f54d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:31.415543 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.415511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c31303-4709-4f8b-86e3-c2288254f54d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04c31303-4709-4f8b-86e3-c2288254f54d" (UID: "04c31303-4709-4f8b-86e3-c2288254f54d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:31.417313 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.417291    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c31303-4709-4f8b-86e3-c2288254f54d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04c31303-4709-4f8b-86e3-c2288254f54d" (UID: "04c31303-4709-4f8b-86e3-c2288254f54d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:11:31.417398 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.417366    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c31303-4709-4f8b-86e3-c2288254f54d-kube-api-access-q498p" (OuterVolumeSpecName: "kube-api-access-q498p") pod "04c31303-4709-4f8b-86e3-c2288254f54d" (UID: "04c31303-4709-4f8b-86e3-c2288254f54d"). InnerVolumeSpecName "kube-api-access-q498p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:11:31.516074 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.516033    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04c31303-4709-4f8b-86e3-c2288254f54d-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:31.516074 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.516070    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:31.516074 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.516080    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04c31303-4709-4f8b-86e3-c2288254f54d-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:31.516074 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.516089    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q498p\" (UniqueName: \"kubernetes.io/projected/04c31303-4709-4f8b-86e3-c2288254f54d-kube-api-access-q498p\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:31.516309 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.516097    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04c31303-4709-4f8b-86e3-c2288254f54d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:31.600508 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.600477    2580 generic.go:356] "Generic (PLEG): container finished" podID="04c31303-4709-4f8b-86e3-c2288254f54d" containerID="29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404" exitCode=0
Sep 17 16:11:31.600622 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.600532    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"
Sep 17 16:11:31.600622 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.600539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" event={"ID":"04c31303-4709-4f8b-86e3-c2288254f54d","Type":"ContainerDied","Data":"29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404"}
Sep 17 16:11:31.600622 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.600618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw" event={"ID":"04c31303-4709-4f8b-86e3-c2288254f54d","Type":"ContainerDied","Data":"788b77319e52b3191955696be22dc69679320f4df82718aecb4a923259ce6ae2"}
Sep 17 16:11:31.600730 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.600646    2580 scope.go:117] "RemoveContainer" containerID="3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005"
Sep 17 16:11:31.609525 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.609509    2580 scope.go:117] "RemoveContainer" containerID="29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404"
Sep 17 16:11:31.617001 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.616985    2580 scope.go:117] "RemoveContainer" containerID="3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637"
Sep 17 16:11:31.623507 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.623485    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"]
Sep 17 16:11:31.624996 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.624980    2580 scope.go:117] "RemoveContainer" containerID="3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005"
Sep 17 16:11:31.625273 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:11:31.625255    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005\": container with ID starting with 3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005 not found: ID does not exist" containerID="3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005"
Sep 17 16:11:31.625319 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.625283    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005"} err="failed to get container status \"3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005\": rpc error: code = NotFound desc = could not find container \"3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005\": container with ID starting with 3f9a7e03f1698366d97321b22a2df600c6df137e6f776c631aa2bac5e0f37005 not found: ID does not exist"
Sep 17 16:11:31.625319 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.625300    2580 scope.go:117] "RemoveContainer" containerID="29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404"
Sep 17 16:11:31.625571 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:11:31.625554    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404\": container with ID starting with 29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404 not found: ID does not exist" containerID="29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404"
Sep 17 16:11:31.625619 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.625578    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404"} err="failed to get container status \"29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404\": rpc error: code = NotFound desc = could not find container \"29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404\": container with ID starting with 29e3f252df759c9b110425665d6ce568d6b6f038050f33bb94309bd3ceee0404 not found: ID does not exist"
Sep 17 16:11:31.625619 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.625594    2580 scope.go:117] "RemoveContainer" containerID="3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637"
Sep 17 16:11:31.625876 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:11:31.625859    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637\": container with ID starting with 3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637 not found: ID does not exist" containerID="3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637"
Sep 17 16:11:31.625925 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.625882    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637"} err="failed to get container status \"3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637\": rpc error: code = NotFound desc = could not find container \"3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637\": container with ID starting with 3b867c0a08bd6aee77cd49a6a6ba99ecbac0a340c0b3148fa126933f737a6637 not found: ID does not exist"
Sep 17 16:11:31.630424 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.630403    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5bb477d776-cvrlw"]
Sep 17 16:11:31.675008 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.674940    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" path="/var/lib/kubelet/pods/04c31303-4709-4f8b-86e3-c2288254f54d/volumes"
Sep 17 16:11:31.919410 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919374    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"]
Sep 17 16:11:31.919677 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919663    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kube-rbac-proxy"
Sep 17 16:11:31.919677 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919677    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kube-rbac-proxy"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919690    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919696    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919703    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kube-rbac-proxy"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919710    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kube-rbac-proxy"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919716    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919721    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919732    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="storage-initializer"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919738    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="storage-initializer"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919744    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="storage-initializer"
Sep 17 16:11:31.919771 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919749    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="storage-initializer"
Sep 17 16:11:31.920109 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919813    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kserve-container"
Sep 17 16:11:31.920109 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919826    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3fca2d3-a468-4b16-8218-e6560650b719" containerName="kube-rbac-proxy"
Sep 17 16:11:31.920109 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919832    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kube-rbac-proxy"
Sep 17 16:11:31.920109 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.919838    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c31303-4709-4f8b-86e3-c2288254f54d" containerName="kserve-container"
Sep 17 16:11:31.924193 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.924177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:31.927993 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.927936    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 16:11:31.928265 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.928251    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 16:11:31.959562 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.959536    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"]
Sep 17 16:11:31.994492 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:31.994456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:32.021037 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.021009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.021037 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.021039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e9a293-dd37-4c98-a832-1358a22a62bf-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.021216 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.021077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.021216 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.021163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e9a293-dd37-4c98-a832-1358a22a62bf-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.021302 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.021212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cljhs\" (UniqueName: \"kubernetes.io/projected/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-api-access-cljhs\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.122011 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.121980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.122011 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.122013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e9a293-dd37-4c98-a832-1358a22a62bf-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.122212 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.122058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.122212 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.122088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e9a293-dd37-4c98-a832-1358a22a62bf-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.122212 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.122115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cljhs\" (UniqueName: \"kubernetes.io/projected/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-api-access-cljhs\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.122513 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.122493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e9a293-dd37-4c98-a832-1358a22a62bf-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.122660 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.122641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.122743 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.122725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.124641 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.124622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e9a293-dd37-4c98-a832-1358a22a62bf-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.129136 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.129109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:32.131144 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.131124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cljhs\" (UniqueName: \"kubernetes.io/projected/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-api-access-cljhs\") pod \"isvc-tensorflow-predictor-547dfc5669-9j8q7\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.233486 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.233421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:32.506590 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.506509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"]
Sep 17 16:11:32.509981 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:11:32.509953    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e9a293_dd37_4c98_a832_1358a22a62bf.slice/crio-7c2a358a5ceb9a4832f9779b2bc6a5d33b6455faa2038ee5df20f389a02049d7 WatchSource:0}: Error finding container 7c2a358a5ceb9a4832f9779b2bc6a5d33b6455faa2038ee5df20f389a02049d7: Status 404 returned error can't find the container with id 7c2a358a5ceb9a4832f9779b2bc6a5d33b6455faa2038ee5df20f389a02049d7
Sep 17 16:11:32.607246 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.607205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" event={"ID":"24e9a293-dd37-4c98-a832-1358a22a62bf","Type":"ContainerStarted","Data":"a5a2b903973ae1b14b9cfd2470164b6204e3b1ed8cd046969a7ad99446fd7974"}
Sep 17 16:11:32.607402 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:32.607281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" event={"ID":"24e9a293-dd37-4c98-a832-1358a22a62bf","Type":"ContainerStarted","Data":"7c2a358a5ceb9a4832f9779b2bc6a5d33b6455faa2038ee5df20f389a02049d7"}
Sep 17 16:11:37.629313 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:37.629279    2580 generic.go:356] "Generic (PLEG): container finished" podID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerID="a5a2b903973ae1b14b9cfd2470164b6204e3b1ed8cd046969a7ad99446fd7974" exitCode=0
Sep 17 16:11:37.629705 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:37.629348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" event={"ID":"24e9a293-dd37-4c98-a832-1358a22a62bf","Type":"ContainerDied","Data":"a5a2b903973ae1b14b9cfd2470164b6204e3b1ed8cd046969a7ad99446fd7974"}
Sep 17 16:11:41.650528 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:41.650501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" event={"ID":"24e9a293-dd37-4c98-a832-1358a22a62bf","Type":"ContainerStarted","Data":"df34d259e5cbd0e2ff4d65af2696948033d543a022918243f56994775e57c060"}
Sep 17 16:11:41.720464 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:41.720435    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"]
Sep 17 16:11:41.720714 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:41.720670    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" containerID="cri-o://893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c" gracePeriod=30
Sep 17 16:11:41.720865 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:41.720778    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kube-rbac-proxy" containerID="cri-o://55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a" gracePeriod=30
Sep 17 16:11:41.761109 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:41.761084    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:42.661849 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:42.661792    2580 generic.go:356] "Generic (PLEG): container finished" podID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerID="55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a" exitCode=2
Sep 17 16:11:42.661849 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:42.661829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" event={"ID":"984d4a56-7f43-4d6a-80b6-b81a33b880fd","Type":"ContainerDied","Data":"55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a"}
Sep 17 16:11:42.664039 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:42.664014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" event={"ID":"24e9a293-dd37-4c98-a832-1358a22a62bf","Type":"ContainerStarted","Data":"f3c600f184e027cbe49c71ecb1ea8fb50e569c11555ad5a0b636814aca803c71"}
Sep 17 16:11:42.664304 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:42.664288    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:42.664432 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:42.664416    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:42.666005 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:42.665983    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:11:42.695539 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:42.695500    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podStartSLOduration=3.385856507 podStartE2EDuration="11.695487616s" totalImagesPullingTime="3.908562954s" totalInitContainerRuntime="4.401068155s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:11:42.694842405 +0000 UTC m=+1987.537205210" watchObservedRunningTime="2026-09-17 16:11:42.695487616 +0000 UTC m=+1987.537850421"
Sep 17 16:11:43.668409 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:43.668366    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:11:44.669905 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:44.669859    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:11:44.672986 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:44.672949    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:11:44.945320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:44.945226    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 16:11:46.669520 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.669499    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:11:46.682036 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.682007    2580 generic.go:356] "Generic (PLEG): container finished" podID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerID="893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c" exitCode=0
Sep 17 16:11:46.682150 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.682055    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"
Sep 17 16:11:46.682150 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.682078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" event={"ID":"984d4a56-7f43-4d6a-80b6-b81a33b880fd","Type":"ContainerDied","Data":"893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c"}
Sep 17 16:11:46.682150 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.682133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k" event={"ID":"984d4a56-7f43-4d6a-80b6-b81a33b880fd","Type":"ContainerDied","Data":"1b56bfd60c9abc0bbde79862b79c127006abe80df853ff79cd40d3ffe0529440"}
Sep 17 16:11:46.682267 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.682160    2580 scope.go:117] "RemoveContainer" containerID="55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a"
Sep 17 16:11:46.689896 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.689880    2580 scope.go:117] "RemoveContainer" containerID="893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c"
Sep 17 16:11:46.697207 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.697191    2580 scope.go:117] "RemoveContainer" containerID="9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f"
Sep 17 16:11:46.705684 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.705660    2580 scope.go:117] "RemoveContainer" containerID="55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a"
Sep 17 16:11:46.706002 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:11:46.705982    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a\": container with ID starting with 55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a not found: ID does not exist" containerID="55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a"
Sep 17 16:11:46.706067 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.706015    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a"} err="failed to get container status \"55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a\": rpc error: code = NotFound desc = could not find container \"55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a\": container with ID starting with 55205a7b6f53ed17eb2203e3e649579c20d71ef87984bb75e465c80c53e3fb9a not found: ID does not exist"
Sep 17 16:11:46.706067 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.706033    2580 scope.go:117] "RemoveContainer" containerID="893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c"
Sep 17 16:11:46.706315 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:11:46.706299    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c\": container with ID starting with 893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c not found: ID does not exist" containerID="893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c"
Sep 17 16:11:46.706356 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.706321    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c"} err="failed to get container status \"893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c\": rpc error: code = NotFound desc = could not find container \"893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c\": container with ID starting with 893282b53f2f58f4c7f0869dec34ad07c811a0c1938935be39866c09ee06db7c not found: ID does not exist"
Sep 17 16:11:46.706356 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.706335    2580 scope.go:117] "RemoveContainer" containerID="9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f"
Sep 17 16:11:46.706595 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:11:46.706577    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f\": container with ID starting with 9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f not found: ID does not exist" containerID="9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f"
Sep 17 16:11:46.706642 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.706601    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f"} err="failed to get container status \"9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f\": rpc error: code = NotFound desc = could not find container \"9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f\": container with ID starting with 9dda65e5a38398e12de4795f9c5da85e845bc7586cd615bd4a92b1a03209440f not found: ID does not exist"
Sep 17 16:11:46.741609 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.741550    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kserve-provision-location\") pod \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") "
Sep 17 16:11:46.741609 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.741597    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-rbac-proxy-sar-config\") pod \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") "
Sep 17 16:11:46.741784 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.741635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/984d4a56-7f43-4d6a-80b6-b81a33b880fd-proxy-tls\") pod \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") "
Sep 17 16:11:46.741784 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.741666    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmnbj\" (UniqueName: \"kubernetes.io/projected/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-api-access-xmnbj\") pod \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") "
Sep 17 16:11:46.741784 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.741687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-cabundle-cert\") pod \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\" (UID: \"984d4a56-7f43-4d6a-80b6-b81a33b880fd\") "
Sep 17 16:11:46.741974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.741918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "984d4a56-7f43-4d6a-80b6-b81a33b880fd" (UID: "984d4a56-7f43-4d6a-80b6-b81a33b880fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:46.742107 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.742065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "984d4a56-7f43-4d6a-80b6-b81a33b880fd" (UID: "984d4a56-7f43-4d6a-80b6-b81a33b880fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:46.742212 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.742119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "984d4a56-7f43-4d6a-80b6-b81a33b880fd" (UID: "984d4a56-7f43-4d6a-80b6-b81a33b880fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:46.743974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.743955    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984d4a56-7f43-4d6a-80b6-b81a33b880fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "984d4a56-7f43-4d6a-80b6-b81a33b880fd" (UID: "984d4a56-7f43-4d6a-80b6-b81a33b880fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:11:46.744075 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.744055    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-api-access-xmnbj" (OuterVolumeSpecName: "kube-api-access-xmnbj") pod "984d4a56-7f43-4d6a-80b6-b81a33b880fd" (UID: "984d4a56-7f43-4d6a-80b6-b81a33b880fd"). InnerVolumeSpecName "kube-api-access-xmnbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:11:46.843026 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.842991    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:46.843026 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.843020    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:46.843026 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.843033    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/984d4a56-7f43-4d6a-80b6-b81a33b880fd-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:46.843248 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.843045    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmnbj\" (UniqueName: \"kubernetes.io/projected/984d4a56-7f43-4d6a-80b6-b81a33b880fd-kube-api-access-xmnbj\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:46.843248 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:46.843056    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/984d4a56-7f43-4d6a-80b6-b81a33b880fd-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:11:47.024145 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.024121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:47.032371 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.032347    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"]
Sep 17 16:11:47.047226 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.047206    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-vws2k"]
Sep 17 16:11:47.086251 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.086215    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"]
Sep 17 16:11:47.086579 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.086562    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container"
Sep 17 16:11:47.086662 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.086583    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container"
Sep 17 16:11:47.086662 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.086604    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kube-rbac-proxy"
Sep 17 16:11:47.086662 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.086622    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kube-rbac-proxy"
Sep 17 16:11:47.086662 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.086641    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="storage-initializer"
Sep 17 16:11:47.086662 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.086650    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="storage-initializer"
Sep 17 16:11:47.086962 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.086741    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kube-rbac-proxy"
Sep 17 16:11:47.086962 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.086757    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" containerName="kserve-container"
Sep 17 16:11:47.091607 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.091587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.099236 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.099219    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 16:11:47.099555 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.099535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 16:11:47.113446 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.113425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"]
Sep 17 16:11:47.145421 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.145387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kb2f\" (UniqueName: \"kubernetes.io/projected/03ffe338-4848-4b7e-8265-044672da1e3d-kube-api-access-7kb2f\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.145539 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.145433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ffe338-4848-4b7e-8265-044672da1e3d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.145539 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.145510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.145632 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.145550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.145632 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.145574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ffe338-4848-4b7e-8265-044672da1e3d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.227668 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.227641    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:47.246885 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.246829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ffe338-4848-4b7e-8265-044672da1e3d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.246885 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.246863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.247023 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.246889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.247023 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:11:47.246985    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 16:11:47.247023 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.247005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ffe338-4848-4b7e-8265-044672da1e3d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.247173 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:11:47.247043    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ffe338-4848-4b7e-8265-044672da1e3d-proxy-tls podName:03ffe338-4848-4b7e-8265-044672da1e3d nodeName:}" failed. No retries permitted until 2026-09-17 16:11:47.747025146 +0000 UTC m=+1992.589387928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/03ffe338-4848-4b7e-8265-044672da1e3d-proxy-tls") pod "isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" (UID: "03ffe338-4848-4b7e-8265-044672da1e3d") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 16:11:47.247173 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.247139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kb2f\" (UniqueName: \"kubernetes.io/projected/03ffe338-4848-4b7e-8265-044672da1e3d-kube-api-access-7kb2f\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.247335 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.247313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ffe338-4848-4b7e-8265-044672da1e3d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.247482 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.247467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.247578 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.247564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.261623 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.258905    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"]
Sep 17 16:11:47.263618 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.263591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.265901 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.265876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kb2f\" (UniqueName: \"kubernetes.io/projected/03ffe338-4848-4b7e-8265-044672da1e3d-kube-api-access-7kb2f\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.273214 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.273192    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 16:11:47.282612 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.282587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"]
Sep 17 16:11:47.324830 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.324791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:47.347519 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.347496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.347600 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.347535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a23f04-6adb-4d84-b528-4ae2f1750e28-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.347600 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.347554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.347674 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.347615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5r9\" (UniqueName: \"kubernetes.io/projected/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-api-access-nc5r9\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.347674 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.347640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a23f04-6adb-4d84-b528-4ae2f1750e28-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.448888 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.448865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.449008 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.448905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a23f04-6adb-4d84-b528-4ae2f1750e28-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.449008 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.448923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.449008 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.448954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5r9\" (UniqueName: \"kubernetes.io/projected/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-api-access-nc5r9\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.449163 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.449073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a23f04-6adb-4d84-b528-4ae2f1750e28-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.449477 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.449458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a23f04-6adb-4d84-b528-4ae2f1750e28-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.449703 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.449682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.449750 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.449682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.451531 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.451514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a23f04-6adb-4d84-b528-4ae2f1750e28-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.463401 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.463374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5r9\" (UniqueName: \"kubernetes.io/projected/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-api-access-nc5r9\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-bgv79\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.574411 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.574341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:47.674210 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.674177    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="984d4a56-7f43-4d6a-80b6-b81a33b880fd" path="/var/lib/kubelet/pods/984d4a56-7f43-4d6a-80b6-b81a33b880fd/volumes"
Sep 17 16:11:47.751683 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.751640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ffe338-4848-4b7e-8265-044672da1e3d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.754156 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.754133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ffe338-4848-4b7e-8265-044672da1e3d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:47.778553 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.778525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:47.863274 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:47.863243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"]
Sep 17 16:11:47.867197 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:11:47.867170    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a23f04_6adb_4d84_b528_4ae2f1750e28.slice/crio-b59be437fbbf64618a0550040937eac8bd4ec2fd38ab9af4ce723aecdf6e147d WatchSource:0}: Error finding container b59be437fbbf64618a0550040937eac8bd4ec2fd38ab9af4ce723aecdf6e147d: Status 404 returned error can't find the container with id b59be437fbbf64618a0550040937eac8bd4ec2fd38ab9af4ce723aecdf6e147d
Sep 17 16:11:48.001739 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:48.001707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:48.294424 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:48.294203    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"]
Sep 17 16:11:48.297290 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:11:48.297259    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ffe338_4848_4b7e_8265_044672da1e3d.slice/crio-e43a7e7bb685a878aae2cf497f91219b494190bf68f2c5a1f66920fc9f4ef39a WatchSource:0}: Error finding container e43a7e7bb685a878aae2cf497f91219b494190bf68f2c5a1f66920fc9f4ef39a: Status 404 returned error can't find the container with id e43a7e7bb685a878aae2cf497f91219b494190bf68f2c5a1f66920fc9f4ef39a
Sep 17 16:11:48.692338 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:48.692298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" event={"ID":"03ffe338-4848-4b7e-8265-044672da1e3d","Type":"ContainerStarted","Data":"505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281"}
Sep 17 16:11:48.692845 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:48.692369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" event={"ID":"03ffe338-4848-4b7e-8265-044672da1e3d","Type":"ContainerStarted","Data":"e43a7e7bb685a878aae2cf497f91219b494190bf68f2c5a1f66920fc9f4ef39a"}
Sep 17 16:11:48.694015 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:48.693986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" event={"ID":"09a23f04-6adb-4d84-b528-4ae2f1750e28","Type":"ContainerStarted","Data":"5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411"}
Sep 17 16:11:48.694120 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:48.694051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" event={"ID":"09a23f04-6adb-4d84-b528-4ae2f1750e28","Type":"ContainerStarted","Data":"b59be437fbbf64618a0550040937eac8bd4ec2fd38ab9af4ce723aecdf6e147d"}
Sep 17 16:11:49.678116 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:49.678091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:49.678603 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:49.678562    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 16:11:52.711215 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:52.711129    2580 generic.go:356] "Generic (PLEG): container finished" podID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerID="5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411" exitCode=0
Sep 17 16:11:52.711215 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:52.711201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" event={"ID":"09a23f04-6adb-4d84-b528-4ae2f1750e28","Type":"ContainerDied","Data":"5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411"}
Sep 17 16:11:52.712931 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:52.712908    2580 generic.go:356] "Generic (PLEG): container finished" podID="03ffe338-4848-4b7e-8265-044672da1e3d" containerID="505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281" exitCode=0
Sep 17 16:11:52.713029 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:52.712972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" event={"ID":"03ffe338-4848-4b7e-8265-044672da1e3d","Type":"ContainerDied","Data":"505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281"}
Sep 17 16:11:53.719523 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.719490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" event={"ID":"09a23f04-6adb-4d84-b528-4ae2f1750e28","Type":"ContainerStarted","Data":"268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd"}
Sep 17 16:11:53.719997 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.719565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" event={"ID":"09a23f04-6adb-4d84-b528-4ae2f1750e28","Type":"ContainerStarted","Data":"3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee"}
Sep 17 16:11:53.719997 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.719931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:53.720113 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.720095    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:53.721497 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.721466    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 16:11:53.721874 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.721854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" event={"ID":"03ffe338-4848-4b7e-8265-044672da1e3d","Type":"ContainerStarted","Data":"bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2"}
Sep 17 16:11:53.721988 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.721903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" event={"ID":"03ffe338-4848-4b7e-8265-044672da1e3d","Type":"ContainerStarted","Data":"bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0"}
Sep 17 16:11:53.722154 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.722137    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:53.722215 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.722173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:53.723102 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.723079    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:11:53.753189 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:53.753137    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podStartSLOduration=2.981798101 podStartE2EDuration="6.753121049s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771322948s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:11:53.752117266 +0000 UTC m=+1998.594480074" watchObservedRunningTime="2026-09-17 16:11:53.753121049 +0000 UTC m=+1998.595483858"
Sep 17 16:11:54.726357 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:54.726313    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 16:11:54.726730 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:54.726423    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:11:59.678854 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:59.678824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:11:59.708982 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:59.708938    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podStartSLOduration=8.677176951 podStartE2EDuration="12.708922097s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.031745146s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:11:53.780836832 +0000 UTC m=+1998.623199634" watchObservedRunningTime="2026-09-17 16:11:59.708922097 +0000 UTC m=+2004.551284901"
Sep 17 16:11:59.731832 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:59.731269    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:11:59.732017 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:59.731917    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 16:11:59.732339 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:59.732314    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:11:59.733052 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:11:59.733022    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:12:09.732508 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:09.732473    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 16:12:09.733685 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:09.733664    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:12:14.420084 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:14.420047    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"]
Sep 17 16:12:14.420577 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:14.420364    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kserve-container" containerID="cri-o://df34d259e5cbd0e2ff4d65af2696948033d543a022918243f56994775e57c060" gracePeriod=30
Sep 17 16:12:14.420756 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:14.420729    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy" containerID="cri-o://f3c600f184e027cbe49c71ecb1ea8fb50e569c11555ad5a0b636814aca803c71" gracePeriod=30
Sep 17 16:12:14.463293 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:14.463265    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:14.674031 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:14.673934    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:12:14.806412 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:14.806381    2580 generic.go:356] "Generic (PLEG): container finished" podID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerID="f3c600f184e027cbe49c71ecb1ea8fb50e569c11555ad5a0b636814aca803c71" exitCode=2
Sep 17 16:12:14.806572 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:14.806455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" event={"ID":"24e9a293-dd37-4c98-a832-1358a22a62bf","Type":"ContainerDied","Data":"f3c600f184e027cbe49c71ecb1ea8fb50e569c11555ad5a0b636814aca803c71"}
Sep 17 16:12:19.673245 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:19.673203    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:12:19.673831 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:19.673594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:19.732692 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:19.732664    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 16:12:19.733062 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:19.733038    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:12:19.764228 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:19.764202    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:24.674158 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:24.674113    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:12:24.674600 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:24.674248    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:12:29.673425 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:29.673389    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:12:29.732559 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:29.732526    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 16:12:29.733792 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:29.733770    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:12:34.673707 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:34.673670    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:12:39.673283 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:39.673241    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:12:39.732178 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:39.732143    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 16:12:39.733314 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:39.733292    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:12:44.673369 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:44.673328    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 16:12:44.933974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:44.933942    2580 generic.go:356] "Generic (PLEG): container finished" podID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerID="df34d259e5cbd0e2ff4d65af2696948033d543a022918243f56994775e57c060" exitCode=137
Sep 17 16:12:44.934143 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:44.934010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" event={"ID":"24e9a293-dd37-4c98-a832-1358a22a62bf","Type":"ContainerDied","Data":"df34d259e5cbd0e2ff4d65af2696948033d543a022918243f56994775e57c060"}
Sep 17 16:12:45.068388 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.068357    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:12:45.117836 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.117784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cljhs\" (UniqueName: \"kubernetes.io/projected/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-api-access-cljhs\") pod \"24e9a293-dd37-4c98-a832-1358a22a62bf\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") "
Sep 17 16:12:45.117963 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.117854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-cabundle-cert\") pod \"24e9a293-dd37-4c98-a832-1358a22a62bf\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") "
Sep 17 16:12:45.117963 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.117890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e9a293-dd37-4c98-a832-1358a22a62bf-kserve-provision-location\") pod \"24e9a293-dd37-4c98-a832-1358a22a62bf\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") "
Sep 17 16:12:45.117963 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.117949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e9a293-dd37-4c98-a832-1358a22a62bf-proxy-tls\") pod \"24e9a293-dd37-4c98-a832-1358a22a62bf\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") "
Sep 17 16:12:45.118114 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.117989    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-rbac-proxy-sar-config\") pod \"24e9a293-dd37-4c98-a832-1358a22a62bf\" (UID: \"24e9a293-dd37-4c98-a832-1358a22a62bf\") "
Sep 17 16:12:45.118263 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.118211    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24e9a293-dd37-4c98-a832-1358a22a62bf" (UID: "24e9a293-dd37-4c98-a832-1358a22a62bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:12:45.118442 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.118414    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24e9a293-dd37-4c98-a832-1358a22a62bf" (UID: "24e9a293-dd37-4c98-a832-1358a22a62bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:12:45.120126 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.120093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e9a293-dd37-4c98-a832-1358a22a62bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24e9a293-dd37-4c98-a832-1358a22a62bf" (UID: "24e9a293-dd37-4c98-a832-1358a22a62bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:12:45.120581 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.120562    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-api-access-cljhs" (OuterVolumeSpecName: "kube-api-access-cljhs") pod "24e9a293-dd37-4c98-a832-1358a22a62bf" (UID: "24e9a293-dd37-4c98-a832-1358a22a62bf"). InnerVolumeSpecName "kube-api-access-cljhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:12:45.129145 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.129124    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e9a293-dd37-4c98-a832-1358a22a62bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24e9a293-dd37-4c98-a832-1358a22a62bf" (UID: "24e9a293-dd37-4c98-a832-1358a22a62bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:12:45.219179 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.219093    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e9a293-dd37-4c98-a832-1358a22a62bf-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:12:45.219179 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.219132    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:12:45.219179 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.219142    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cljhs\" (UniqueName: \"kubernetes.io/projected/24e9a293-dd37-4c98-a832-1358a22a62bf-kube-api-access-cljhs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:12:45.219179 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.219151    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e9a293-dd37-4c98-a832-1358a22a62bf-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:12:45.219179 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.219159    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e9a293-dd37-4c98-a832-1358a22a62bf-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:12:45.939867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.939760    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"
Sep 17 16:12:45.939867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.939761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7" event={"ID":"24e9a293-dd37-4c98-a832-1358a22a62bf","Type":"ContainerDied","Data":"7c2a358a5ceb9a4832f9779b2bc6a5d33b6455faa2038ee5df20f389a02049d7"}
Sep 17 16:12:45.940370 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.939883    2580 scope.go:117] "RemoveContainer" containerID="f3c600f184e027cbe49c71ecb1ea8fb50e569c11555ad5a0b636814aca803c71"
Sep 17 16:12:45.948023 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.948003    2580 scope.go:117] "RemoveContainer" containerID="df34d259e5cbd0e2ff4d65af2696948033d543a022918243f56994775e57c060"
Sep 17 16:12:45.955124 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.955110    2580 scope.go:117] "RemoveContainer" containerID="a5a2b903973ae1b14b9cfd2470164b6204e3b1ed8cd046969a7ad99446fd7974"
Sep 17 16:12:45.964085 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.964050    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"]
Sep 17 16:12:45.966104 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:45.966076    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9j8q7"]
Sep 17 16:12:47.674843 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:47.674795    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" path="/var/lib/kubelet/pods/24e9a293-dd37-4c98-a832-1358a22a62bf/volumes"
Sep 17 16:12:49.732042 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:49.731998    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 16:12:49.733212 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:49.733189    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:12:57.685271 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:57.685196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:59.731989 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:59.731951    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 16:12:59.733169 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:12:59.733144    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:13:01.231273 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:01.231244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:06.344487 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.344454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:06.381291 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.381255    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"]
Sep 17 16:13:06.381628 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.381610    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy"
Sep 17 16:13:06.381713 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.381630    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy"
Sep 17 16:13:06.381713 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.381651    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kserve-container"
Sep 17 16:13:06.381713 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.381659    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kserve-container"
Sep 17 16:13:06.381713 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.381692    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="storage-initializer"
Sep 17 16:13:06.381713 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.381701    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="storage-initializer"
Sep 17 16:13:06.381999 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.381778    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kube-rbac-proxy"
Sep 17 16:13:06.381999 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.381793    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e9a293-dd37-4c98-a832-1358a22a62bf" containerName="kserve-container"
Sep 17 16:13:06.386388 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.386368    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.388498 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.388471    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 16:13:06.388603 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.388518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 16:13:06.397571 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.397546    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"]
Sep 17 16:13:06.501383 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.501359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e74c61d9-46de-4379-8b4b-0d3c76d72354-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.501527 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.501396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74c61d9-46de-4379-8b4b-0d3c76d72354-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.501527 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.501426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rs9d\" (UniqueName: \"kubernetes.io/projected/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-api-access-5rs9d\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.501527 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.501462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.501527 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.501484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.602723 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.602647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e74c61d9-46de-4379-8b4b-0d3c76d72354-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.602723 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.602692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74c61d9-46de-4379-8b4b-0d3c76d72354-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.602956 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.602737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rs9d\" (UniqueName: \"kubernetes.io/projected/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-api-access-5rs9d\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.602956 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.602776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.602956 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.602835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.603204 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.603178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e74c61d9-46de-4379-8b4b-0d3c76d72354-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.603493 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.603474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.603554 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.603494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.605294 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.605275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74c61d9-46de-4379-8b4b-0d3c76d72354-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.611163 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.611140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rs9d\" (UniqueName: \"kubernetes.io/projected/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-api-access-5rs9d\") pod \"isvc-triton-predictor-5d7db58676-zwxtg\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.627371 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.627350    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:06.696685 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.696663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:13:06.972570 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:06.972495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"]
Sep 17 16:13:06.974849 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:13:06.974821    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74c61d9_46de_4379_8b4b_0d3c76d72354.slice/crio-b2ecd57c9baa58adef27f2d3aba101e354fd1db6f6121624c2376d58974c25b4 WatchSource:0}: Error finding container b2ecd57c9baa58adef27f2d3aba101e354fd1db6f6121624c2376d58974c25b4: Status 404 returned error can't find the container with id b2ecd57c9baa58adef27f2d3aba101e354fd1db6f6121624c2376d58974c25b4
Sep 17 16:13:07.026077 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:07.026042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" event={"ID":"e74c61d9-46de-4379-8b4b-0d3c76d72354","Type":"ContainerStarted","Data":"b2ecd57c9baa58adef27f2d3aba101e354fd1db6f6121624c2376d58974c25b4"}
Sep 17 16:13:07.675300 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:07.675276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:13:07.827925 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:07.827892    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"]
Sep 17 16:13:07.828145 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:07.828110    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container" containerID="cri-o://3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee" gracePeriod=30
Sep 17 16:13:07.828210 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:07.828164    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kube-rbac-proxy" containerID="cri-o://268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd" gracePeriod=30
Sep 17 16:13:07.828962 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:07.828944    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:13:08.031645 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:08.031602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" event={"ID":"e74c61d9-46de-4379-8b4b-0d3c76d72354","Type":"ContainerStarted","Data":"990383a5724a9462f9134d2f2fde47da2177ed9ddc2752729eb63bf660241aab"}
Sep 17 16:13:08.033826 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:08.033787    2580 generic.go:356] "Generic (PLEG): container finished" podID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerID="268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd" exitCode=2
Sep 17 16:13:08.033938 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:08.033824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" event={"ID":"09a23f04-6adb-4d84-b528-4ae2f1750e28","Type":"ContainerDied","Data":"268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd"}
Sep 17 16:13:09.727249 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:09.727200    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 16:13:12.051104 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.051070    2580 generic.go:356] "Generic (PLEG): container finished" podID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerID="990383a5724a9462f9134d2f2fde47da2177ed9ddc2752729eb63bf660241aab" exitCode=0
Sep 17 16:13:12.051460 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.051144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" event={"ID":"e74c61d9-46de-4379-8b4b-0d3c76d72354","Type":"ContainerDied","Data":"990383a5724a9462f9134d2f2fde47da2177ed9ddc2752729eb63bf660241aab"}
Sep 17 16:13:12.052333 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.052318    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:13:12.264351 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.264327    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:13:12.449900 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.449873    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-rbac-proxy-sar-config\") pod \"09a23f04-6adb-4d84-b528-4ae2f1750e28\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") "
Sep 17 16:13:12.450085 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.449917    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-cabundle-cert\") pod \"09a23f04-6adb-4d84-b528-4ae2f1750e28\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") "
Sep 17 16:13:12.450085 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.449937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a23f04-6adb-4d84-b528-4ae2f1750e28-kserve-provision-location\") pod \"09a23f04-6adb-4d84-b528-4ae2f1750e28\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") "
Sep 17 16:13:12.450085 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.449972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc5r9\" (UniqueName: \"kubernetes.io/projected/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-api-access-nc5r9\") pod \"09a23f04-6adb-4d84-b528-4ae2f1750e28\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") "
Sep 17 16:13:12.450085 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.450053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a23f04-6adb-4d84-b528-4ae2f1750e28-proxy-tls\") pod \"09a23f04-6adb-4d84-b528-4ae2f1750e28\" (UID: \"09a23f04-6adb-4d84-b528-4ae2f1750e28\") "
Sep 17 16:13:12.450312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.450262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a23f04-6adb-4d84-b528-4ae2f1750e28-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09a23f04-6adb-4d84-b528-4ae2f1750e28" (UID: "09a23f04-6adb-4d84-b528-4ae2f1750e28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:13:12.450312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.450263    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09a23f04-6adb-4d84-b528-4ae2f1750e28" (UID: "09a23f04-6adb-4d84-b528-4ae2f1750e28"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:13:12.450312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.450288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09a23f04-6adb-4d84-b528-4ae2f1750e28" (UID: "09a23f04-6adb-4d84-b528-4ae2f1750e28"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:13:12.452258 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.452231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a23f04-6adb-4d84-b528-4ae2f1750e28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09a23f04-6adb-4d84-b528-4ae2f1750e28" (UID: "09a23f04-6adb-4d84-b528-4ae2f1750e28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:13:12.452368 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.452288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-api-access-nc5r9" (OuterVolumeSpecName: "kube-api-access-nc5r9") pod "09a23f04-6adb-4d84-b528-4ae2f1750e28" (UID: "09a23f04-6adb-4d84-b528-4ae2f1750e28"). InnerVolumeSpecName "kube-api-access-nc5r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:13:12.551075 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.551045    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:12.551075 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.551072    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a23f04-6adb-4d84-b528-4ae2f1750e28-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:12.551244 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.551084    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc5r9\" (UniqueName: \"kubernetes.io/projected/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-api-access-nc5r9\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:12.551244 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.551093    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a23f04-6adb-4d84-b528-4ae2f1750e28-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:12.551244 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:12.551101    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a23f04-6adb-4d84-b528-4ae2f1750e28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:13.061783 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.061742    2580 generic.go:356] "Generic (PLEG): container finished" podID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerID="3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee" exitCode=0
Sep 17 16:13:13.062251 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.061842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" event={"ID":"09a23f04-6adb-4d84-b528-4ae2f1750e28","Type":"ContainerDied","Data":"3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee"}
Sep 17 16:13:13.062251 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.061903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79" event={"ID":"09a23f04-6adb-4d84-b528-4ae2f1750e28","Type":"ContainerDied","Data":"b59be437fbbf64618a0550040937eac8bd4ec2fd38ab9af4ce723aecdf6e147d"}
Sep 17 16:13:13.062251 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.061954    2580 scope.go:117] "RemoveContainer" containerID="268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd"
Sep 17 16:13:13.062251 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.062151    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"
Sep 17 16:13:13.084842 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.084706    2580 scope.go:117] "RemoveContainer" containerID="3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee"
Sep 17 16:13:13.096326 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.096278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"]
Sep 17 16:13:13.100422 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.100277    2580 scope.go:117] "RemoveContainer" containerID="5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411"
Sep 17 16:13:13.101526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.101474    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-bgv79"]
Sep 17 16:13:13.118526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.118436    2580 scope.go:117] "RemoveContainer" containerID="268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd"
Sep 17 16:13:13.119216 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:13:13.119163    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd\": container with ID starting with 268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd not found: ID does not exist" containerID="268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd"
Sep 17 16:13:13.119316 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.119204    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd"} err="failed to get container status \"268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd\": rpc error: code = NotFound desc = could not find container \"268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd\": container with ID starting with 268089ac9fa8130388243790bd08872645749ff3c6f3391dc46964096e002bbd not found: ID does not exist"
Sep 17 16:13:13.119316 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.119256    2580 scope.go:117] "RemoveContainer" containerID="3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee"
Sep 17 16:13:13.122109 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:13:13.122059    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee\": container with ID starting with 3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee not found: ID does not exist" containerID="3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee"
Sep 17 16:13:13.122109 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.122097    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee"} err="failed to get container status \"3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee\": rpc error: code = NotFound desc = could not find container \"3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee\": container with ID starting with 3b41186f31ac1ae3dbfad3092053d8402d93069f2f6ab80aaa4974612f0193ee not found: ID does not exist"
Sep 17 16:13:13.122327 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.122121    2580 scope.go:117] "RemoveContainer" containerID="5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411"
Sep 17 16:13:13.123194 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:13:13.123137    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411\": container with ID starting with 5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411 not found: ID does not exist" containerID="5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411"
Sep 17 16:13:13.123194 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.123167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411"} err="failed to get container status \"5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411\": rpc error: code = NotFound desc = could not find container \"5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411\": container with ID starting with 5958be842c630e0fe42b0ff3fa0800beed152a9dccf426d87c99e32012568411 not found: ID does not exist"
Sep 17 16:13:13.680008 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:13.679971    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" path="/var/lib/kubelet/pods/09a23f04-6adb-4d84-b528-4ae2f1750e28/volumes"
Sep 17 16:13:16.970837 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:16.969334    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"]
Sep 17 16:13:16.970837 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:16.969640    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" containerID="cri-o://bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0" gracePeriod=30
Sep 17 16:13:16.970837 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:16.970053    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kube-rbac-proxy" containerID="cri-o://bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2" gracePeriod=30
Sep 17 16:13:16.994316 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:16.994211    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:17.008344 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.006435    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"]
Sep 17 16:13:17.008344 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.006904    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="storage-initializer"
Sep 17 16:13:17.008344 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.006921    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="storage-initializer"
Sep 17 16:13:17.008344 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.006949    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container"
Sep 17 16:13:17.008344 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.006957    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container"
Sep 17 16:13:17.008344 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.006981    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kube-rbac-proxy"
Sep 17 16:13:17.008344 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.006989    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kube-rbac-proxy"
Sep 17 16:13:17.008344 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.007059    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kserve-container"
Sep 17 16:13:17.008344 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.007073    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a23f04-6adb-4d84-b528-4ae2f1750e28" containerName="kube-rbac-proxy"
Sep 17 16:13:17.012299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.012270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.014902 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.014784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 16:13:17.026236 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.026170    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"]
Sep 17 16:13:17.093280 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.093032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.093280 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.093148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkdz\" (UniqueName: \"kubernetes.io/projected/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-api-access-zpkdz\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.093280 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.093178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.093280 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.093220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.093631 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.093312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.194195 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.193777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.194195 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.193895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.194195 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.193936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkdz\" (UniqueName: \"kubernetes.io/projected/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-api-access-zpkdz\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.194195 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.193961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.194195 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.193993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.194601 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.194574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.194952 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.194909    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.195457 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.195398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.197941 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.197902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.204678 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.204646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkdz\" (UniqueName: \"kubernetes.io/projected/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-api-access-zpkdz\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-9vph6\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.331883 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.331707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:17.670401 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:17.670018    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 16:13:19.727420 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:19.727378    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 17 16:13:19.818550 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:19.818472    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"]
Sep 17 16:13:20.130722 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:20.130171    2580 generic.go:356] "Generic (PLEG): container finished" podID="03ffe338-4848-4b7e-8265-044672da1e3d" containerID="bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2" exitCode=2
Sep 17 16:13:20.130722 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:20.130238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" event={"ID":"03ffe338-4848-4b7e-8265-044672da1e3d","Type":"ContainerDied","Data":"bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2"}
Sep 17 16:13:23.023446 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.023363    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:13:23.052189 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.052119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ffe338-4848-4b7e-8265-044672da1e3d-kserve-provision-location\") pod \"03ffe338-4848-4b7e-8265-044672da1e3d\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") "
Sep 17 16:13:23.053334 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.052916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ffe338-4848-4b7e-8265-044672da1e3d-proxy-tls\") pod \"03ffe338-4848-4b7e-8265-044672da1e3d\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") "
Sep 17 16:13:23.053680 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.053495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kb2f\" (UniqueName: \"kubernetes.io/projected/03ffe338-4848-4b7e-8265-044672da1e3d-kube-api-access-7kb2f\") pod \"03ffe338-4848-4b7e-8265-044672da1e3d\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") "
Sep 17 16:13:23.054206 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.054147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-kube-rbac-proxy-sar-config\") pod \"03ffe338-4848-4b7e-8265-044672da1e3d\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") "
Sep 17 16:13:23.054923 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.054718    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03ffe338-4848-4b7e-8265-044672da1e3d" (UID: "03ffe338-4848-4b7e-8265-044672da1e3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:13:23.055828 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.055227    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-cabundle-cert\") pod \"03ffe338-4848-4b7e-8265-044672da1e3d\" (UID: \"03ffe338-4848-4b7e-8265-044672da1e3d\") "
Sep 17 16:13:23.055828 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.055501    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:23.056144 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.056110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03ffe338-4848-4b7e-8265-044672da1e3d" (UID: "03ffe338-4848-4b7e-8265-044672da1e3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:13:23.058106 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.058079    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ffe338-4848-4b7e-8265-044672da1e3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03ffe338-4848-4b7e-8265-044672da1e3d" (UID: "03ffe338-4848-4b7e-8265-044672da1e3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:13:23.060896 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.060359    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ffe338-4848-4b7e-8265-044672da1e3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03ffe338-4848-4b7e-8265-044672da1e3d" (UID: "03ffe338-4848-4b7e-8265-044672da1e3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:13:23.060896 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.060437    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ffe338-4848-4b7e-8265-044672da1e3d-kube-api-access-7kb2f" (OuterVolumeSpecName: "kube-api-access-7kb2f") pod "03ffe338-4848-4b7e-8265-044672da1e3d" (UID: "03ffe338-4848-4b7e-8265-044672da1e3d"). InnerVolumeSpecName "kube-api-access-7kb2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:13:23.156307 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.156137    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kb2f\" (UniqueName: \"kubernetes.io/projected/03ffe338-4848-4b7e-8265-044672da1e3d-kube-api-access-7kb2f\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:23.156307 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.156167    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03ffe338-4848-4b7e-8265-044672da1e3d-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:23.156307 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.156182    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03ffe338-4848-4b7e-8265-044672da1e3d-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:23.156307 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.156196    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03ffe338-4848-4b7e-8265-044672da1e3d-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:13:23.161502 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.156949    2580 generic.go:356] "Generic (PLEG): container finished" podID="03ffe338-4848-4b7e-8265-044672da1e3d" containerID="bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0" exitCode=0
Sep 17 16:13:23.161502 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.157131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" event={"ID":"03ffe338-4848-4b7e-8265-044672da1e3d","Type":"ContainerDied","Data":"bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0"}
Sep 17 16:13:23.161502 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.157193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f" event={"ID":"03ffe338-4848-4b7e-8265-044672da1e3d","Type":"ContainerDied","Data":"e43a7e7bb685a878aae2cf497f91219b494190bf68f2c5a1f66920fc9f4ef39a"}
Sep 17 16:13:23.161502 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.157234    2580 scope.go:117] "RemoveContainer" containerID="bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2"
Sep 17 16:13:23.161502 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.157410    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"
Sep 17 16:13:23.169247 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.168398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" event={"ID":"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354","Type":"ContainerStarted","Data":"deafec397eec2d6c23d4c77c4498feab526dfe9f7c5457ee1e934a2c553e6e0e"}
Sep 17 16:13:23.169247 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.168461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" event={"ID":"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354","Type":"ContainerStarted","Data":"1ed6a4a163edfa801fc3c3631ddc8f919b48c3a5f02aa3d11458331553f0b499"}
Sep 17 16:13:23.179717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.179534    2580 scope.go:117] "RemoveContainer" containerID="bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0"
Sep 17 16:13:23.193868 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.193843    2580 scope.go:117] "RemoveContainer" containerID="505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281"
Sep 17 16:13:23.216554 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.216495    2580 scope.go:117] "RemoveContainer" containerID="bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2"
Sep 17 16:13:23.219604 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:13:23.217516    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2\": container with ID starting with bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2 not found: ID does not exist" containerID="bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2"
Sep 17 16:13:23.219604 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.217550    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2"} err="failed to get container status \"bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2\": rpc error: code = NotFound desc = could not find container \"bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2\": container with ID starting with bf28012d0cd783f57643896f39a218214436601a5277c2f0bbb372b8cf8486d2 not found: ID does not exist"
Sep 17 16:13:23.219604 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.217575    2580 scope.go:117] "RemoveContainer" containerID="bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0"
Sep 17 16:13:23.221974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.220823    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"]
Sep 17 16:13:23.221974 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:13:23.221162    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0\": container with ID starting with bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0 not found: ID does not exist" containerID="bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0"
Sep 17 16:13:23.221974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.221194    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0"} err="failed to get container status \"bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0\": rpc error: code = NotFound desc = could not find container \"bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0\": container with ID starting with bec7ded9226d74b1b83a8ca5bd056fb38914fe6710c5a2e481929d54541c10b0 not found: ID does not exist"
Sep 17 16:13:23.221974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.221218    2580 scope.go:117] "RemoveContainer" containerID="505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281"
Sep 17 16:13:23.221974 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:13:23.221915    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281\": container with ID starting with 505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281 not found: ID does not exist" containerID="505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281"
Sep 17 16:13:23.221974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.221942    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281"} err="failed to get container status \"505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281\": rpc error: code = NotFound desc = could not find container \"505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281\": container with ID starting with 505ade1182ab2d7af6fa58ad9da4ce1b83f51a3d1f05e6e1e05b673bd4bcf281 not found: ID does not exist"
Sep 17 16:13:23.224764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.224729    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-fb5f8487c-rjr6f"]
Sep 17 16:13:23.687814 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:23.687318    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" path="/var/lib/kubelet/pods/03ffe338-4848-4b7e-8265-044672da1e3d/volumes"
Sep 17 16:13:28.207121 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:28.207071    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerID="deafec397eec2d6c23d4c77c4498feab526dfe9f7c5457ee1e934a2c553e6e0e" exitCode=0
Sep 17 16:13:28.207608 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:28.207127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" event={"ID":"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354","Type":"ContainerDied","Data":"deafec397eec2d6c23d4c77c4498feab526dfe9f7c5457ee1e934a2c553e6e0e"}
Sep 17 16:13:29.215958 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:29.215914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" event={"ID":"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354","Type":"ContainerStarted","Data":"0892a6aa6eb3dd4d4ac462d329eba26d4e0b812ee0cc02a9022ddafb064f2b50"}
Sep 17 16:13:29.216421 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:29.215998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" event={"ID":"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354","Type":"ContainerStarted","Data":"953ba66f0df000b3ec86af043a7ebe78e5a58b77d1ce62be9bacb45f0c272b38"}
Sep 17 16:13:29.216837 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:29.216785    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:29.216963 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:29.216861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:29.217727 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:29.217699    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:13:29.243096 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:29.243041    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podStartSLOduration=8.649004839 podStartE2EDuration="13.243011971s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.594007132s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:13:29.24097168 +0000 UTC m=+2094.083334508" watchObservedRunningTime="2026-09-17 16:13:29.243011971 +0000 UTC m=+2094.085374779"
Sep 17 16:13:30.221831 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:30.221506    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:13:35.228537 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:35.227724    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:13:35.229202 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:35.229171    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:13:35.738349 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:35.738319    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:13:35.743849 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:35.743745    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:13:45.228309 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:45.228238    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:13:55.228973 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:13:55.228927    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:14:05.229033 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:14:05.228985    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:14:15.228739 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:14:15.228681    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:14:25.228769 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:14:25.228722    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:14:35.229203 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:14:35.229106    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:14:36.297924 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:14:36.297888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:36.384124 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:14:36.384079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:45.228413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:14:45.228374    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:14:55.228600 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:14:55.228561    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:15:01.670632 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:01.670584    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 16:15:08.679935 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:08.679897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" event={"ID":"e74c61d9-46de-4379-8b4b-0d3c76d72354","Type":"ContainerStarted","Data":"808035bae153a80e4b6ba7b3363f767f2505351f5cb326a6de225fbdca729bb8"}
Sep 17 16:15:08.680353 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:08.679957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" event={"ID":"e74c61d9-46de-4379-8b4b-0d3c76d72354","Type":"ContainerStarted","Data":"857f64902a2b474a20ff18d9154fdcd5471370c45f93866dc82618bc92200829"}
Sep 17 16:15:08.680353 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:08.679982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:15:08.680353 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:08.680001    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:15:08.681485 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:08.681456    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 16:15:08.710454 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:08.710412    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" podStartSLOduration=2.674609272 podStartE2EDuration="2m2.710396657s" totalImagesPullingTime="1m55.835166428s" totalInitContainerRuntime="4.200620957s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:15:08.708466733 +0000 UTC m=+2193.550829536" watchObservedRunningTime="2026-09-17 16:15:08.710396657 +0000 UTC m=+2193.552759460"
Sep 17 16:15:09.683501 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:09.683459    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 16:15:11.674547 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:11.674520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:15:14.688897 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:14.688867    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:15:14.689727 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:14.689708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:15:17.173787 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:17.173752    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"]
Sep 17 16:15:17.189274 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:17.174536    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" containerID="cri-o://953ba66f0df000b3ec86af043a7ebe78e5a58b77d1ce62be9bacb45f0c272b38" gracePeriod=30
Sep 17 16:15:17.189274 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:17.174788    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kube-rbac-proxy" containerID="cri-o://0892a6aa6eb3dd4d4ac462d329eba26d4e0b812ee0cc02a9022ddafb064f2b50" gracePeriod=30
Sep 17 16:15:17.204755 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:17.204726    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:17.714505 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:17.714464    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerID="0892a6aa6eb3dd4d4ac462d329eba26d4e0b812ee0cc02a9022ddafb064f2b50" exitCode=2
Sep 17 16:15:17.714673 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:17.714512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" event={"ID":"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354","Type":"ContainerDied","Data":"0892a6aa6eb3dd4d4ac462d329eba26d4e0b812ee0cc02a9022ddafb064f2b50"}
Sep 17 16:15:18.720181 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:18.720147    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"]
Sep 17 16:15:18.720574 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:18.720369    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kserve-container" containerID="cri-o://857f64902a2b474a20ff18d9154fdcd5471370c45f93866dc82618bc92200829" gracePeriod=30
Sep 17 16:15:18.720574 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:18.720412    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kube-rbac-proxy" containerID="cri-o://808035bae153a80e4b6ba7b3363f767f2505351f5cb326a6de225fbdca729bb8" gracePeriod=30
Sep 17 16:15:18.771756 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:18.771708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:19.684667 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:19.684625    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 16:15:19.724783 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:19.724756    2580 generic.go:356] "Generic (PLEG): container finished" podID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerID="808035bae153a80e4b6ba7b3363f767f2505351f5cb326a6de225fbdca729bb8" exitCode=2
Sep 17 16:15:19.725144 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:19.724822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" event={"ID":"e74c61d9-46de-4379-8b4b-0d3c76d72354","Type":"ContainerDied","Data":"808035bae153a80e4b6ba7b3363f767f2505351f5cb326a6de225fbdca729bb8"}
Sep 17 16:15:20.222149 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:20.222110    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 16:15:21.734147 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.734116    2580 generic.go:356] "Generic (PLEG): container finished" podID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerID="857f64902a2b474a20ff18d9154fdcd5471370c45f93866dc82618bc92200829" exitCode=0
Sep 17 16:15:21.734542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.734180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" event={"ID":"e74c61d9-46de-4379-8b4b-0d3c76d72354","Type":"ContainerDied","Data":"857f64902a2b474a20ff18d9154fdcd5471370c45f93866dc82618bc92200829"}
Sep 17 16:15:21.736145 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.736120    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerID="953ba66f0df000b3ec86af043a7ebe78e5a58b77d1ce62be9bacb45f0c272b38" exitCode=0
Sep 17 16:15:21.736284 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.736192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" event={"ID":"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354","Type":"ContainerDied","Data":"953ba66f0df000b3ec86af043a7ebe78e5a58b77d1ce62be9bacb45f0c272b38"}
Sep 17 16:15:21.736284 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.736239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" event={"ID":"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354","Type":"ContainerDied","Data":"1ed6a4a163edfa801fc3c3631ddc8f919b48c3a5f02aa3d11458331553f0b499"}
Sep 17 16:15:21.736284 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.736259    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ed6a4a163edfa801fc3c3631ddc8f919b48c3a5f02aa3d11458331553f0b499"
Sep 17 16:15:21.736877 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.736860    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:15:21.757358 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.757340    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:15:21.786917 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.786894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-cabundle-cert\") pod \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") "
Sep 17 16:15:21.787076 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.786938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-proxy-tls\") pod \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") "
Sep 17 16:15:21.787076 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.786968    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-rbac-proxy-sar-config\") pod \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") "
Sep 17 16:15:21.787076 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.787002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpkdz\" (UniqueName: \"kubernetes.io/projected/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-api-access-zpkdz\") pod \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") "
Sep 17 16:15:21.787076 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.787063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kserve-provision-location\") pod \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\" (UID: \"ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354\") "
Sep 17 16:15:21.787369 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.787259    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" (UID: "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:15:21.787369 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.787359    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" (UID: "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:15:21.787502 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.787443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" (UID: "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:15:21.789145 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.789125    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" (UID: "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:15:21.789295 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.789280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-api-access-zpkdz" (OuterVolumeSpecName: "kube-api-access-zpkdz") pod "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" (UID: "ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354"). InnerVolumeSpecName "kube-api-access-zpkdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:15:21.888257 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74c61d9-46de-4379-8b4b-0d3c76d72354-proxy-tls\") pod \"e74c61d9-46de-4379-8b4b-0d3c76d72354\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") "
Sep 17 16:15:21.888402 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e74c61d9-46de-4379-8b4b-0d3c76d72354-kserve-provision-location\") pod \"e74c61d9-46de-4379-8b4b-0d3c76d72354\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") "
Sep 17 16:15:21.888402 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888281    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-rbac-proxy-sar-config\") pod \"e74c61d9-46de-4379-8b4b-0d3c76d72354\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") "
Sep 17 16:15:21.888402 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rs9d\" (UniqueName: \"kubernetes.io/projected/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-api-access-5rs9d\") pod \"e74c61d9-46de-4379-8b4b-0d3c76d72354\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") "
Sep 17 16:15:21.888402 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-cabundle-cert\") pod \"e74c61d9-46de-4379-8b4b-0d3c76d72354\" (UID: \"e74c61d9-46de-4379-8b4b-0d3c76d72354\") "
Sep 17 16:15:21.888587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888524    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:21.888587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888543    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:21.888587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888558    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:21.888587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888570    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:21.888587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888585    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkdz\" (UniqueName: \"kubernetes.io/projected/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354-kube-api-access-zpkdz\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:21.888874 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e74c61d9-46de-4379-8b4b-0d3c76d72354" (UID: "e74c61d9-46de-4379-8b4b-0d3c76d72354"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:15:21.888874 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74c61d9-46de-4379-8b4b-0d3c76d72354-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e74c61d9-46de-4379-8b4b-0d3c76d72354" (UID: "e74c61d9-46de-4379-8b4b-0d3c76d72354"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:15:21.888874 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.888829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e74c61d9-46de-4379-8b4b-0d3c76d72354" (UID: "e74c61d9-46de-4379-8b4b-0d3c76d72354"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:15:21.890363 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.890346    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74c61d9-46de-4379-8b4b-0d3c76d72354-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e74c61d9-46de-4379-8b4b-0d3c76d72354" (UID: "e74c61d9-46de-4379-8b4b-0d3c76d72354"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:15:21.890421 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.890396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-api-access-5rs9d" (OuterVolumeSpecName: "kube-api-access-5rs9d") pod "e74c61d9-46de-4379-8b4b-0d3c76d72354" (UID: "e74c61d9-46de-4379-8b4b-0d3c76d72354"). InnerVolumeSpecName "kube-api-access-5rs9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:15:21.989256 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.989187    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e74c61d9-46de-4379-8b4b-0d3c76d72354-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:21.989256 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.989210    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e74c61d9-46de-4379-8b4b-0d3c76d72354-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:21.989256 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.989221    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:21.989256 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.989229    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rs9d\" (UniqueName: \"kubernetes.io/projected/e74c61d9-46de-4379-8b4b-0d3c76d72354-kube-api-access-5rs9d\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:21.989256 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:21.989237    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e74c61d9-46de-4379-8b4b-0d3c76d72354-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.275460 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.275365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:22.350354 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350326    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"]
Sep 17 16:15:22.350610 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350598    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="storage-initializer"
Sep 17 16:15:22.350657 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350613    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="storage-initializer"
Sep 17 16:15:22.350657 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350623    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="storage-initializer"
Sep 17 16:15:22.350657 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350629    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="storage-initializer"
Sep 17 16:15:22.350657 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350637    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kube-rbac-proxy"
Sep 17 16:15:22.350657 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350643    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kube-rbac-proxy"
Sep 17 16:15:22.350657 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350654    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kube-rbac-proxy"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350660    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kube-rbac-proxy"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350667    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kserve-container"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350672    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kserve-container"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350680    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="storage-initializer"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350685    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="storage-initializer"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350692    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350697    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350706    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350710    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350718    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kube-rbac-proxy"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kube-rbac-proxy"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350765    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kube-rbac-proxy"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350770    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kserve-container"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350777    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" containerName="kube-rbac-proxy"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350783    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350788    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kube-rbac-proxy"
Sep 17 16:15:22.350867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.350820    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ffe338-4848-4b7e-8265-044672da1e3d" containerName="kserve-container"
Sep 17 16:15:22.372434 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.372406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:22.385256 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.385221    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"]
Sep 17 16:15:22.385392 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.385352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.388826 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.388791    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 16:15:22.388941 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.388823    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 16:15:22.445980 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.442831    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"]
Sep 17 16:15:22.465012 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.464988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"]
Sep 17 16:15:22.465126 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.465079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.467321 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.467300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 16:15:22.492575 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.492550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlvsj\" (UniqueName: \"kubernetes.io/projected/09469b89-babd-447d-b55f-709dace765b3-kube-api-access-xlvsj\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.492667 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.492583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09469b89-babd-447d-b55f-709dace765b3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.492667 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.492610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.492667 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.492658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.492775 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.492735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09469b89-babd-447d-b55f-709dace765b3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.594058 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.593994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzj84\" (UniqueName: \"kubernetes.io/projected/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-api-access-rzj84\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.594058 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.594058 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.594238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.594238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1de6f68-56e8-45fd-a120-3189dbcd432e-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.594238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1de6f68-56e8-45fd-a120-3189dbcd432e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.594238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09469b89-babd-447d-b55f-709dace765b3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.594413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlvsj\" (UniqueName: \"kubernetes.io/projected/09469b89-babd-447d-b55f-709dace765b3-kube-api-access-xlvsj\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.594413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09469b89-babd-447d-b55f-709dace765b3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.594413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.594413 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:15:22.594380    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 16:15:22.594595 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:15:22.594440    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09469b89-babd-447d-b55f-709dace765b3-proxy-tls podName:09469b89-babd-447d-b55f-709dace765b3 nodeName:}" failed. No retries permitted until 2026-09-17 16:15:23.094420238 +0000 UTC m=+2207.936783030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/09469b89-babd-447d-b55f-709dace765b3-proxy-tls") pod "isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" (UID: "09469b89-babd-447d-b55f-709dace765b3") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 16:15:22.594703 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.594739 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09469b89-babd-447d-b55f-709dace765b3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.594826 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.594787    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.600679 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.600659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:22.611326 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.611307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlvsj\" (UniqueName: \"kubernetes.io/projected/09469b89-babd-447d-b55f-709dace765b3-kube-api-access-xlvsj\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:22.670822 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.670778    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: i/o timeout"
Sep 17 16:15:22.695634 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.695612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1de6f68-56e8-45fd-a120-3189dbcd432e-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.695714 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.695644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1de6f68-56e8-45fd-a120-3189dbcd432e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.695753 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.695730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.695785 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.695759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzj84\" (UniqueName: \"kubernetes.io/projected/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-api-access-rzj84\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.695856 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.695783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.696101 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.696081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1de6f68-56e8-45fd-a120-3189dbcd432e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.696398 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.696383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.696442 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.696386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.698015 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.697997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1de6f68-56e8-45fd-a120-3189dbcd432e-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.709100 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.709066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzj84\" (UniqueName: \"kubernetes.io/projected/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-api-access-rzj84\") pod \"isvc-canary-stop-predictor-556d599d48-pcrdx\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.741963 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.741944    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"
Sep 17 16:15:22.742240 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.741960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg" event={"ID":"e74c61d9-46de-4379-8b4b-0d3c76d72354","Type":"ContainerDied","Data":"b2ecd57c9baa58adef27f2d3aba101e354fd1db6f6121624c2376d58974c25b4"}
Sep 17 16:15:22.742240 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.742028    2580 scope.go:117] "RemoveContainer" containerID="808035bae153a80e4b6ba7b3363f767f2505351f5cb326a6de225fbdca729bb8"
Sep 17 16:15:22.742240 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.741971    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"
Sep 17 16:15:22.750671 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.750652    2580 scope.go:117] "RemoveContainer" containerID="857f64902a2b474a20ff18d9154fdcd5471370c45f93866dc82618bc92200829"
Sep 17 16:15:22.757994 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.757978    2580 scope.go:117] "RemoveContainer" containerID="990383a5724a9462f9134d2f2fde47da2177ed9ddc2752729eb63bf660241aab"
Sep 17 16:15:22.768264 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.768245    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"]
Sep 17 16:15:22.772518 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.772499    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-9vph6"]
Sep 17 16:15:22.774131 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.774117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:22.785509 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.785489    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"]
Sep 17 16:15:22.789526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.789503    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-zwxtg"]
Sep 17 16:15:22.924964 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:22.924926    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:23.098336 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.098309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09469b89-babd-447d-b55f-709dace765b3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:23.100681 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.100654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09469b89-babd-447d-b55f-709dace765b3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:23.132956 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.132933    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"]
Sep 17 16:15:23.135434 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:15:23.135402    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1de6f68_56e8_45fd_a120_3189dbcd432e.slice/crio-16698c3c16493c4b45324c44500ff06036aab022c99e5f2fc506b0555f18e8c7 WatchSource:0}: Error finding container 16698c3c16493c4b45324c44500ff06036aab022c99e5f2fc506b0555f18e8c7: Status 404 returned error can't find the container with id 16698c3c16493c4b45324c44500ff06036aab022c99e5f2fc506b0555f18e8c7
Sep 17 16:15:23.296918 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.296892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:23.644565 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.644500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"]
Sep 17 16:15:23.646975 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:15:23.646941    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09469b89_babd_447d_b55f_709dace765b3.slice/crio-062cd35d1ff14ea412f865dd70cc8094125033f7ca9b94226c147000500960d7 WatchSource:0}: Error finding container 062cd35d1ff14ea412f865dd70cc8094125033f7ca9b94226c147000500960d7: Status 404 returned error can't find the container with id 062cd35d1ff14ea412f865dd70cc8094125033f7ca9b94226c147000500960d7
Sep 17 16:15:23.675943 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.675916    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e74c61d9-46de-4379-8b4b-0d3c76d72354" path="/var/lib/kubelet/pods/e74c61d9-46de-4379-8b4b-0d3c76d72354/volumes"
Sep 17 16:15:23.676478 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.676459    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354" path="/var/lib/kubelet/pods/ee0cac6c-c0ec-4e5e-b63a-1e4ff082e354/volumes"
Sep 17 16:15:23.750596 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.750564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" event={"ID":"09469b89-babd-447d-b55f-709dace765b3","Type":"ContainerStarted","Data":"3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728"}
Sep 17 16:15:23.751073 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.750620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" event={"ID":"09469b89-babd-447d-b55f-709dace765b3","Type":"ContainerStarted","Data":"062cd35d1ff14ea412f865dd70cc8094125033f7ca9b94226c147000500960d7"}
Sep 17 16:15:23.752407 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.752375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" event={"ID":"f1de6f68-56e8-45fd-a120-3189dbcd432e","Type":"ContainerStarted","Data":"d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d"}
Sep 17 16:15:23.752518 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.752438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" event={"ID":"f1de6f68-56e8-45fd-a120-3189dbcd432e","Type":"ContainerStarted","Data":"16698c3c16493c4b45324c44500ff06036aab022c99e5f2fc506b0555f18e8c7"}
Sep 17 16:15:23.881566 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:23.881530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:24.179968 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:24.179935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:27.768781 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:27.768747    2580 generic.go:356] "Generic (PLEG): container finished" podID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerID="d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d" exitCode=0
Sep 17 16:15:27.769135 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:27.768815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" event={"ID":"f1de6f68-56e8-45fd-a120-3189dbcd432e","Type":"ContainerDied","Data":"d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d"}
Sep 17 16:15:27.770747 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:27.770724    2580 generic.go:356] "Generic (PLEG): container finished" podID="09469b89-babd-447d-b55f-709dace765b3" containerID="3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728" exitCode=0
Sep 17 16:15:27.770880 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:27.770759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" event={"ID":"09469b89-babd-447d-b55f-709dace765b3","Type":"ContainerDied","Data":"3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728"}
Sep 17 16:15:28.777893 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.777855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" event={"ID":"f1de6f68-56e8-45fd-a120-3189dbcd432e","Type":"ContainerStarted","Data":"d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e"}
Sep 17 16:15:28.778380 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.777931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" event={"ID":"f1de6f68-56e8-45fd-a120-3189dbcd432e","Type":"ContainerStarted","Data":"bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf"}
Sep 17 16:15:28.778380 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.778331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:28.778530 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.778388    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:28.779982 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.779951    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:15:28.780374 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.780342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" event={"ID":"09469b89-babd-447d-b55f-709dace765b3","Type":"ContainerStarted","Data":"9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56"}
Sep 17 16:15:28.780432 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.780402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" event={"ID":"09469b89-babd-447d-b55f-709dace765b3","Type":"ContainerStarted","Data":"f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d"}
Sep 17 16:15:28.780677 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.780656    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:28.780770 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.780709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:28.781666 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.781642    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:15:28.803680 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.803632    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podStartSLOduration=2.339552472 podStartE2EDuration="6.803619132s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.46406666s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:15:28.80188252 +0000 UTC m=+2213.644245335" watchObservedRunningTime="2026-09-17 16:15:28.803619132 +0000 UTC m=+2213.645981937"
Sep 17 16:15:28.825807 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:28.825768    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podStartSLOduration=2.8015014799999998 podStartE2EDuration="6.825756342s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.024254862s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:15:28.824569002 +0000 UTC m=+2213.666931804" watchObservedRunningTime="2026-09-17 16:15:28.825756342 +0000 UTC m=+2213.668119146"
Sep 17 16:15:29.784838 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:29.784768    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:15:29.784838 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:29.784819    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:15:34.790323 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:34.790289    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:15:34.790845 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:34.790614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:15:34.790901 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:34.790838    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:15:34.791384 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:34.791365    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:15:44.791826 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:44.791769    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:15:44.792289 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:44.791769    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:15:54.790933 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:54.790886    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:15:54.791322 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:15:54.791305    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:16:04.791001 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:04.790922    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:16:04.791370 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:04.791343    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:16:14.791034 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:14.790997    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:16:14.791471 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:14.791338    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:16:22.521217 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:22.521184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:16:24.791216 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:24.791179    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:16:24.791697 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:24.791278    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:16:25.902500 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:25.902470    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:16:34.790852 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:34.790790    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:16:34.791241 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:34.791221    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:16:44.791698 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:44.791669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:16:44.792187 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:44.791774    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:16:52.393685 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:52.393646    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"]
Sep 17 16:16:52.394209 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:52.393971    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" containerID="cri-o://f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d" gracePeriod=30
Sep 17 16:16:52.394209 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:52.394067    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kube-rbac-proxy" containerID="cri-o://9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56" gracePeriod=30
Sep 17 16:16:52.421678 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:52.421639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:16:52.487020 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:52.486985    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"]
Sep 17 16:16:52.487267 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:52.487242    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" containerID="cri-o://bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf" gracePeriod=30
Sep 17 16:16:52.487315 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:52.487280    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kube-rbac-proxy" containerID="cri-o://d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e" gracePeriod=30
Sep 17 16:16:53.129561 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:53.129524    2580 generic.go:356] "Generic (PLEG): container finished" podID="09469b89-babd-447d-b55f-709dace765b3" containerID="9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56" exitCode=2
Sep 17 16:16:53.129883 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:53.129606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" event={"ID":"09469b89-babd-447d-b55f-709dace765b3","Type":"ContainerDied","Data":"9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56"}
Sep 17 16:16:53.131740 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:53.131715    2580 generic.go:356] "Generic (PLEG): container finished" podID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerID="d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e" exitCode=2
Sep 17 16:16:53.131916 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:53.131751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" event={"ID":"f1de6f68-56e8-45fd-a120-3189dbcd432e","Type":"ContainerDied","Data":"d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e"}
Sep 17 16:16:54.064159 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:54.064119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:16:54.785921 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:54.785880    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 16:16:54.786144 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:54.785880    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 16:16:54.791521 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:54.791478    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 16:16:54.791662 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:54.791478    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 16:16:57.861401 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.861371    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:16:57.864453 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.864429    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:16:57.958088 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.957984    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-cabundle-cert\") pod \"09469b89-babd-447d-b55f-709dace765b3\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") "
Sep 17 16:16:57.958088 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzj84\" (UniqueName: \"kubernetes.io/projected/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-api-access-rzj84\") pod \"f1de6f68-56e8-45fd-a120-3189dbcd432e\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") "
Sep 17 16:16:57.958088 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-kube-rbac-proxy-sar-config\") pod \"09469b89-babd-447d-b55f-709dace765b3\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") "
Sep 17 16:16:57.958357 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958098    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-rbac-proxy-sar-config\") pod \"f1de6f68-56e8-45fd-a120-3189dbcd432e\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") "
Sep 17 16:16:57.958357 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958138    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-cabundle-cert\") pod \"f1de6f68-56e8-45fd-a120-3189dbcd432e\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") "
Sep 17 16:16:57.958357 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09469b89-babd-447d-b55f-709dace765b3-kserve-provision-location\") pod \"09469b89-babd-447d-b55f-709dace765b3\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") "
Sep 17 16:16:57.958357 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlvsj\" (UniqueName: \"kubernetes.io/projected/09469b89-babd-447d-b55f-709dace765b3-kube-api-access-xlvsj\") pod \"09469b89-babd-447d-b55f-709dace765b3\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") "
Sep 17 16:16:57.958357 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1de6f68-56e8-45fd-a120-3189dbcd432e-kserve-provision-location\") pod \"f1de6f68-56e8-45fd-a120-3189dbcd432e\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") "
Sep 17 16:16:57.958357 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958289    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09469b89-babd-447d-b55f-709dace765b3-proxy-tls\") pod \"09469b89-babd-447d-b55f-709dace765b3\" (UID: \"09469b89-babd-447d-b55f-709dace765b3\") "
Sep 17 16:16:57.958357 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958312    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1de6f68-56e8-45fd-a120-3189dbcd432e-proxy-tls\") pod \"f1de6f68-56e8-45fd-a120-3189dbcd432e\" (UID: \"f1de6f68-56e8-45fd-a120-3189dbcd432e\") "
Sep 17 16:16:57.958717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958475    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09469b89-babd-447d-b55f-709dace765b3" (UID: "09469b89-babd-447d-b55f-709dace765b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:16:57.958717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09469b89-babd-447d-b55f-709dace765b3" (UID: "09469b89-babd-447d-b55f-709dace765b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:16:57.958717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1de6f68-56e8-45fd-a120-3189dbcd432e" (UID: "f1de6f68-56e8-45fd-a120-3189dbcd432e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:16:57.958717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f1de6f68-56e8-45fd-a120-3189dbcd432e" (UID: "f1de6f68-56e8-45fd-a120-3189dbcd432e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:16:57.958717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958563    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:57.958717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958580    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:57.958717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958594    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09469b89-babd-447d-b55f-709dace765b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:57.959080 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958763    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1de6f68-56e8-45fd-a120-3189dbcd432e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1de6f68-56e8-45fd-a120-3189dbcd432e" (UID: "f1de6f68-56e8-45fd-a120-3189dbcd432e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:16:57.959080 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.958908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09469b89-babd-447d-b55f-709dace765b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09469b89-babd-447d-b55f-709dace765b3" (UID: "09469b89-babd-447d-b55f-709dace765b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:16:57.960586 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.960555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-api-access-rzj84" (OuterVolumeSpecName: "kube-api-access-rzj84") pod "f1de6f68-56e8-45fd-a120-3189dbcd432e" (UID: "f1de6f68-56e8-45fd-a120-3189dbcd432e"). InnerVolumeSpecName "kube-api-access-rzj84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:16:57.960586 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.960570    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1de6f68-56e8-45fd-a120-3189dbcd432e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1de6f68-56e8-45fd-a120-3189dbcd432e" (UID: "f1de6f68-56e8-45fd-a120-3189dbcd432e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:16:57.961156 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.961133    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09469b89-babd-447d-b55f-709dace765b3-kube-api-access-xlvsj" (OuterVolumeSpecName: "kube-api-access-xlvsj") pod "09469b89-babd-447d-b55f-709dace765b3" (UID: "09469b89-babd-447d-b55f-709dace765b3"). InnerVolumeSpecName "kube-api-access-xlvsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:16:57.961242 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:57.961128    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09469b89-babd-447d-b55f-709dace765b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09469b89-babd-447d-b55f-709dace765b3" (UID: "09469b89-babd-447d-b55f-709dace765b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:16:58.058987 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.058951    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09469b89-babd-447d-b55f-709dace765b3-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:58.058987 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.058981    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1de6f68-56e8-45fd-a120-3189dbcd432e-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:58.058987 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.058990    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzj84\" (UniqueName: \"kubernetes.io/projected/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-api-access-rzj84\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:58.059228 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.058999    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1de6f68-56e8-45fd-a120-3189dbcd432e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:58.059228 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.059008    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09469b89-babd-447d-b55f-709dace765b3-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:58.059228 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.059017    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlvsj\" (UniqueName: \"kubernetes.io/projected/09469b89-babd-447d-b55f-709dace765b3-kube-api-access-xlvsj\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:58.059228 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.059025    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1de6f68-56e8-45fd-a120-3189dbcd432e-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:16:58.153997 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.153964    2580 generic.go:356] "Generic (PLEG): container finished" podID="09469b89-babd-447d-b55f-709dace765b3" containerID="f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d" exitCode=0
Sep 17 16:16:58.154200 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.154037    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"
Sep 17 16:16:58.154200 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.154053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" event={"ID":"09469b89-babd-447d-b55f-709dace765b3","Type":"ContainerDied","Data":"f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d"}
Sep 17 16:16:58.154200 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.154122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l" event={"ID":"09469b89-babd-447d-b55f-709dace765b3","Type":"ContainerDied","Data":"062cd35d1ff14ea412f865dd70cc8094125033f7ca9b94226c147000500960d7"}
Sep 17 16:16:58.154200 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.154156    2580 scope.go:117] "RemoveContainer" containerID="9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56"
Sep 17 16:16:58.156169 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.156142    2580 generic.go:356] "Generic (PLEG): container finished" podID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerID="bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf" exitCode=0
Sep 17 16:16:58.156301 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.156194    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"
Sep 17 16:16:58.156301 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.156222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" event={"ID":"f1de6f68-56e8-45fd-a120-3189dbcd432e","Type":"ContainerDied","Data":"bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf"}
Sep 17 16:16:58.156301 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.156288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx" event={"ID":"f1de6f68-56e8-45fd-a120-3189dbcd432e","Type":"ContainerDied","Data":"16698c3c16493c4b45324c44500ff06036aab022c99e5f2fc506b0555f18e8c7"}
Sep 17 16:16:58.163273 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.163244    2580 scope.go:117] "RemoveContainer" containerID="f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d"
Sep 17 16:16:58.171982 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.171953    2580 scope.go:117] "RemoveContainer" containerID="3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728"
Sep 17 16:16:58.180046 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.180015    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"]
Sep 17 16:16:58.180781 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.180760    2580 scope.go:117] "RemoveContainer" containerID="9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56"
Sep 17 16:16:58.181253 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:16:58.181234    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56\": container with ID starting with 9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56 not found: ID does not exist" containerID="9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56"
Sep 17 16:16:58.181307 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.181265    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56"} err="failed to get container status \"9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56\": rpc error: code = NotFound desc = could not find container \"9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56\": container with ID starting with 9503c42302a415bbf7a8080fa1569413d6dc23f2e17facf011b583af8a95ae56 not found: ID does not exist"
Sep 17 16:16:58.181307 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.181286    2580 scope.go:117] "RemoveContainer" containerID="f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d"
Sep 17 16:16:58.181538 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:16:58.181524    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d\": container with ID starting with f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d not found: ID does not exist" containerID="f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d"
Sep 17 16:16:58.181583 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.181541    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d"} err="failed to get container status \"f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d\": rpc error: code = NotFound desc = could not find container \"f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d\": container with ID starting with f9984dc70d8279521232cd353123a0c84876d73ba9bc2ed281c083808d010c8d not found: ID does not exist"
Sep 17 16:16:58.181583 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.181554    2580 scope.go:117] "RemoveContainer" containerID="3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728"
Sep 17 16:16:58.181779 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:16:58.181764    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728\": container with ID starting with 3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728 not found: ID does not exist" containerID="3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728"
Sep 17 16:16:58.181843 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.181784    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728"} err="failed to get container status \"3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728\": rpc error: code = NotFound desc = could not find container \"3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728\": container with ID starting with 3a7928551d8a6619aa7c8c1c89624a5eacc7491486e557ce9001b64c751f3728 not found: ID does not exist"
Sep 17 16:16:58.181843 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.181820    2580 scope.go:117] "RemoveContainer" containerID="d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e"
Sep 17 16:16:58.184840 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.184814    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-gs78l"]
Sep 17 16:16:58.190261 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.190239    2580 scope.go:117] "RemoveContainer" containerID="bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf"
Sep 17 16:16:58.195701 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.195674    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"]
Sep 17 16:16:58.199364 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.199343    2580 scope.go:117] "RemoveContainer" containerID="d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d"
Sep 17 16:16:58.199722 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.199700    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-pcrdx"]
Sep 17 16:16:58.207702 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.207681    2580 scope.go:117] "RemoveContainer" containerID="d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e"
Sep 17 16:16:58.208083 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:16:58.208062    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e\": container with ID starting with d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e not found: ID does not exist" containerID="d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e"
Sep 17 16:16:58.208196 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.208095    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e"} err="failed to get container status \"d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e\": rpc error: code = NotFound desc = could not find container \"d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e\": container with ID starting with d254d8c04085bfc56585315727c2acb197c8283342d58c0d8cf0c5a44df0758e not found: ID does not exist"
Sep 17 16:16:58.208196 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.208122    2580 scope.go:117] "RemoveContainer" containerID="bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf"
Sep 17 16:16:58.208429 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:16:58.208412    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf\": container with ID starting with bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf not found: ID does not exist" containerID="bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf"
Sep 17 16:16:58.208465 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.208435    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf"} err="failed to get container status \"bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf\": rpc error: code = NotFound desc = could not find container \"bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf\": container with ID starting with bd50b9d81defa96d341f8fba61ef3f65a71a8078fcaf684e5a7d83420bd363cf not found: ID does not exist"
Sep 17 16:16:58.208465 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.208452    2580 scope.go:117] "RemoveContainer" containerID="d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d"
Sep 17 16:16:58.208684 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:16:58.208664    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d\": container with ID starting with d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d not found: ID does not exist" containerID="d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d"
Sep 17 16:16:58.208744 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:58.208693    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d"} err="failed to get container status \"d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d\": rpc error: code = NotFound desc = could not find container \"d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d\": container with ID starting with d336794d9361f86b1a0f80d2da3973d8486d2e442633de127e3f671e8835652d not found: ID does not exist"
Sep 17 16:16:59.674906 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:59.674871    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09469b89-babd-447d-b55f-709dace765b3" path="/var/lib/kubelet/pods/09469b89-babd-447d-b55f-709dace765b3/volumes"
Sep 17 16:16:59.675359 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:16:59.675344    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" path="/var/lib/kubelet/pods/f1de6f68-56e8-45fd-a120-3189dbcd432e/volumes"
Sep 17 16:17:02.617491 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.617432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:02.679728 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.679684    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"]
Sep 17 16:17:02.680099 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680081    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container"
Sep 17 16:17:02.680185 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680102    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container"
Sep 17 16:17:02.680185 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680122    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="storage-initializer"
Sep 17 16:17:02.680185 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680131    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="storage-initializer"
Sep 17 16:17:02.680185 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680143    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kube-rbac-proxy"
Sep 17 16:17:02.680185 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680152    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kube-rbac-proxy"
Sep 17 16:17:02.680185 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680166    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kube-rbac-proxy"
Sep 17 16:17:02.680185 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680175    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kube-rbac-proxy"
Sep 17 16:17:02.680526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680193    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container"
Sep 17 16:17:02.680526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680201    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container"
Sep 17 16:17:02.680526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680211    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="storage-initializer"
Sep 17 16:17:02.680526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680219    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="storage-initializer"
Sep 17 16:17:02.680526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680298    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kserve-container"
Sep 17 16:17:02.680526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680310    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kube-rbac-proxy"
Sep 17 16:17:02.680526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680319    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09469b89-babd-447d-b55f-709dace765b3" containerName="kube-rbac-proxy"
Sep 17 16:17:02.680526 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.680331    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1de6f68-56e8-45fd-a120-3189dbcd432e" containerName="kserve-container"
Sep 17 16:17:02.684035 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.684011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.686458 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.686433    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:17:02.686976 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.686953    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 16:17:02.687076 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.686953    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 16:17:02.687076 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.686961    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:17:02.687211 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.687102    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:17:02.687335 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.687317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:17:02.687411 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.687369    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:17:02.697557 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.697523    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"]
Sep 17 16:17:02.797509 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.797472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.797509 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.797512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.797726 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.797541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29jj9\" (UniqueName: \"kubernetes.io/projected/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-api-access-29jj9\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.797726 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.797633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.797896 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.797753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.801858 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.801831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:02.898977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.898884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.898977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.898927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.898977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.898947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29jj9\" (UniqueName: \"kubernetes.io/projected/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-api-access-29jj9\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.898977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.898974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.899253 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.899051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.899492 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.899462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.899645 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.899619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.899712 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.899695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.901689 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.901659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.908513 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.908479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29jj9\" (UniqueName: \"kubernetes.io/projected/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-api-access-29jj9\") pod \"isvc-lightgbm-predictor-54f85ddf88-rl5dc\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:02.994370 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:02.994333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:03.279900 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:03.279872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"]
Sep 17 16:17:03.282498 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:17:03.282459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c90e3f2_0bfa_459b_8c96_e8ac9ddc66f4.slice/crio-de2ddb084f89c8bb62809d1dbf1f4abbacc27e3bd9e60850a73e47667c270bb3 WatchSource:0}: Error finding container de2ddb084f89c8bb62809d1dbf1f4abbacc27e3bd9e60850a73e47667c270bb3: Status 404 returned error can't find the container with id de2ddb084f89c8bb62809d1dbf1f4abbacc27e3bd9e60850a73e47667c270bb3
Sep 17 16:17:04.182641 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:04.182601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" event={"ID":"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4","Type":"ContainerStarted","Data":"a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74"}
Sep 17 16:17:04.183083 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:04.182674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" event={"ID":"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4","Type":"ContainerStarted","Data":"de2ddb084f89c8bb62809d1dbf1f4abbacc27e3bd9e60850a73e47667c270bb3"}
Sep 17 16:17:04.493188 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:04.493089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:08.202028 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:08.201986    2580 generic.go:356] "Generic (PLEG): container finished" podID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerID="a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74" exitCode=0
Sep 17 16:17:08.202435 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:08.202064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" event={"ID":"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4","Type":"ContainerDied","Data":"a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74"}
Sep 17 16:17:09.649866 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:09.649770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:09.912956 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:09.912857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:15.238499 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:15.238456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" event={"ID":"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4","Type":"ContainerStarted","Data":"c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494"}
Sep 17 16:17:15.238924 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:15.238523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" event={"ID":"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4","Type":"ContainerStarted","Data":"475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c"}
Sep 17 16:17:15.238924 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:15.238708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:15.279447 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:15.279378    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podStartSLOduration=2.276198122 podStartE2EDuration="13.279360352s" totalImagesPullingTime="6.85850144s" totalInitContainerRuntime="4.14466079s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:15.271316644 +0000 UTC m=+2320.113679450" watchObservedRunningTime="2026-09-17 16:17:15.279360352 +0000 UTC m=+2320.121723157"
Sep 17 16:17:16.242976 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:16.242941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:16.244567 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:16.244527    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:17:17.247335 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:17.247294    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:17:22.257726 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:22.257688    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:17:22.258389 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:22.258361    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:17:32.258667 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:32.258572    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:17:42.259173 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:42.259125    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:17:52.258875 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:52.258833    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:17:59.973325 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:17:59.973284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:02.259067 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:02.259026    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:18:04.882910 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:04.882871    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:05.615743 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:05.615716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:12.259172 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:12.259129    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:18:22.258815 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:22.258771    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:18:32.258302 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:32.258260    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:18:35.768510 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:35.768479    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:18:35.775911 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:35.775885    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:18:42.259320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:42.259272    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:18:42.669545 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:42.669498    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 16:18:52.669970 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:52.669934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:18:55.037356 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:18:55.037326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:00.152971 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.152892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:00.200172 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.200134    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"]
Sep 17 16:19:00.203444 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.203413    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.205273 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.205254    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 16:19:00.205370 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.205273    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:19:00.215542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.215514    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"]
Sep 17 16:19:00.326021 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.325982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.326021 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.326020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.326241 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.326039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.326241 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.326103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbcbm\" (UniqueName: \"kubernetes.io/projected/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-api-access-gbcbm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.326241 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.326130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac5406e-d16c-43b0-bf52-6a82d71d294f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.426911 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.426768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.426911 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.426859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.426911 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.426886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.427217 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.426945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbcbm\" (UniqueName: \"kubernetes.io/projected/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-api-access-gbcbm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.427217 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.426980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac5406e-d16c-43b0-bf52-6a82d71d294f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.427217 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:19:00.427142    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 17 16:19:00.427371 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:19:00.427226    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ac5406e-d16c-43b0-bf52-6a82d71d294f-proxy-tls podName:7ac5406e-d16c-43b0-bf52-6a82d71d294f nodeName:}" failed. No retries permitted until 2026-09-17 16:19:00.927202313 +0000 UTC m=+2425.769565110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7ac5406e-d16c-43b0-bf52-6a82d71d294f-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" (UID: "7ac5406e-d16c-43b0-bf52-6a82d71d294f") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 17 16:19:00.427371 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.427266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.427556 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.427538    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.427598 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.427543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.436196 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.436169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbcbm\" (UniqueName: \"kubernetes.io/projected/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-api-access-gbcbm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.626979 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.626947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:00.932278 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.932241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac5406e-d16c-43b0-bf52-6a82d71d294f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:00.934853 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:00.934823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac5406e-d16c-43b0-bf52-6a82d71d294f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tl8h4\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:01.113550 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:01.113510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:01.396443 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:01.396415    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"]
Sep 17 16:19:01.398744 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:19:01.398716    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac5406e_d16c_43b0_bf52_6a82d71d294f.slice/crio-dfa743af3b89d2b47e8d09bcfdda4b4e6560b8d962e88bcf8f8a194103192948 WatchSource:0}: Error finding container dfa743af3b89d2b47e8d09bcfdda4b4e6560b8d962e88bcf8f8a194103192948: Status 404 returned error can't find the container with id dfa743af3b89d2b47e8d09bcfdda4b4e6560b8d962e88bcf8f8a194103192948
Sep 17 16:19:01.662373 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:01.662266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" event={"ID":"7ac5406e-d16c-43b0-bf52-6a82d71d294f","Type":"ContainerStarted","Data":"bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f"}
Sep 17 16:19:01.662517 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:01.662390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" event={"ID":"7ac5406e-d16c-43b0-bf52-6a82d71d294f","Type":"ContainerStarted","Data":"dfa743af3b89d2b47e8d09bcfdda4b4e6560b8d962e88bcf8f8a194103192948"}
Sep 17 16:19:03.501697 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:03.501655    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"]
Sep 17 16:19:03.502173 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:03.501934    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container" containerID="cri-o://475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c" gracePeriod=30
Sep 17 16:19:03.502173 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:03.501987    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kube-rbac-proxy" containerID="cri-o://c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494" gracePeriod=30
Sep 17 16:19:03.507012 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:03.506986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:03.673612 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:03.673580    2580 generic.go:356] "Generic (PLEG): container finished" podID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerID="c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494" exitCode=2
Sep 17 16:19:03.674539 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:03.674517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" event={"ID":"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4","Type":"ContainerDied","Data":"c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494"}
Sep 17 16:19:05.683349 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:05.683314    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerID="bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f" exitCode=0
Sep 17 16:19:05.683763 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:05.683389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" event={"ID":"7ac5406e-d16c-43b0-bf52-6a82d71d294f","Type":"ContainerDied","Data":"bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f"}
Sep 17 16:19:05.685443 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:05.685410    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:19:07.248400 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:07.248352    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 16:19:08.718380 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:08.718341    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:08.838352 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:08.838303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:10.369493 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.369465    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:19:10.518279 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.518246    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-cabundle-cert\") pod \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") "
Sep 17 16:19:10.518453 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.518289    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29jj9\" (UniqueName: \"kubernetes.io/projected/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-api-access-29jj9\") pod \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") "
Sep 17 16:19:10.518453 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.518375    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-proxy-tls\") pod \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") "
Sep 17 16:19:10.518453 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.518395    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kserve-provision-location\") pod \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") "
Sep 17 16:19:10.518608 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.518523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-rbac-proxy-sar-config\") pod \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\" (UID: \"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4\") "
Sep 17 16:19:10.518674 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.518655    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" (UID: "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:19:10.518841 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.518784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" (UID: "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:19:10.518974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.518877    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:19:10.518974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.518894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" (UID: "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:19:10.520690 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.520667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-api-access-29jj9" (OuterVolumeSpecName: "kube-api-access-29jj9") pod "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" (UID: "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4"). InnerVolumeSpecName "kube-api-access-29jj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:19:10.520690 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.520666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" (UID: "6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:19:10.620227 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.620194    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29jj9\" (UniqueName: \"kubernetes.io/projected/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-api-access-29jj9\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:19:10.620227 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.620222    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:19:10.620227 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.620234    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:19:10.620467 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.620244    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:19:10.707404 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.707358    2580 generic.go:356] "Generic (PLEG): container finished" podID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerID="475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c" exitCode=0
Sep 17 16:19:10.707598 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.707438    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"
Sep 17 16:19:10.707598 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.707451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" event={"ID":"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4","Type":"ContainerDied","Data":"475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c"}
Sep 17 16:19:10.707598 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.707512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc" event={"ID":"6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4","Type":"ContainerDied","Data":"de2ddb084f89c8bb62809d1dbf1f4abbacc27e3bd9e60850a73e47667c270bb3"}
Sep 17 16:19:10.707598 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.707542    2580 scope.go:117] "RemoveContainer" containerID="c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494"
Sep 17 16:19:10.716039 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.716018    2580 scope.go:117] "RemoveContainer" containerID="475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c"
Sep 17 16:19:10.725444 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.725275    2580 scope.go:117] "RemoveContainer" containerID="a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74"
Sep 17 16:19:10.734906 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.734865    2580 scope.go:117] "RemoveContainer" containerID="c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494"
Sep 17 16:19:10.735263 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:19:10.735237    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494\": container with ID starting with c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494 not found: ID does not exist" containerID="c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494"
Sep 17 16:19:10.735368 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.735272    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494"} err="failed to get container status \"c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494\": rpc error: code = NotFound desc = could not find container \"c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494\": container with ID starting with c379e51c063821ca8b5b74c6a5c6f8c473ea19db7bb4157539fcb0c839e41494 not found: ID does not exist"
Sep 17 16:19:10.735368 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.735297    2580 scope.go:117] "RemoveContainer" containerID="475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c"
Sep 17 16:19:10.735662 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:19:10.735642    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c\": container with ID starting with 475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c not found: ID does not exist" containerID="475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c"
Sep 17 16:19:10.735717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.735669    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c"} err="failed to get container status \"475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c\": rpc error: code = NotFound desc = could not find container \"475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c\": container with ID starting with 475b889093d74bbe82344f7c08e234313eeba0a99d2718bfd22fb1745dc57e8c not found: ID does not exist"
Sep 17 16:19:10.735717 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.735685    2580 scope.go:117] "RemoveContainer" containerID="a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74"
Sep 17 16:19:10.736016 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:19:10.735995    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74\": container with ID starting with a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74 not found: ID does not exist" containerID="a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74"
Sep 17 16:19:10.736104 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.736026    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74"} err="failed to get container status \"a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74\": rpc error: code = NotFound desc = could not find container \"a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74\": container with ID starting with a0079c5738637a113e7f7065d10a8253eb8ea19a3a8066d23f24cb6f63d8ec74 not found: ID does not exist"
Sep 17 16:19:10.746248 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.746194    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"]
Sep 17 16:19:10.753467 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:10.753444    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-rl5dc"]
Sep 17 16:19:11.674269 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:11.674233    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" path="/var/lib/kubelet/pods/6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4/volumes"
Sep 17 16:19:17.876523 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:17.876482    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:26.782949 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:26.782901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" event={"ID":"7ac5406e-d16c-43b0-bf52-6a82d71d294f","Type":"ContainerStarted","Data":"0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360"}
Sep 17 16:19:26.783376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:26.782994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" event={"ID":"7ac5406e-d16c-43b0-bf52-6a82d71d294f","Type":"ContainerStarted","Data":"8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4"}
Sep 17 16:19:26.783376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:26.783286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:26.783510 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:26.783461    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:26.784754 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:26.784724    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:19:26.802893 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:26.802838    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podStartSLOduration=2.413960405 podStartE2EDuration="26.802793467s" totalImagesPullingTime="20.557472557s" totalInitContainerRuntime="3.831360505s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:19:26.801878896 +0000 UTC m=+2451.644241702" watchObservedRunningTime="2026-09-17 16:19:26.802793467 +0000 UTC m=+2451.645156272"
Sep 17 16:19:27.787116 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:27.787069    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:19:32.792266 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:32.792229    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:19:32.792824 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:32.792776    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:19:35.897465 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:35.897428    2580 scope.go:117] "RemoveContainer" containerID="953ba66f0df000b3ec86af043a7ebe78e5a58b77d1ce62be9bacb45f0c272b38"
Sep 17 16:19:35.905540 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:35.905516    2580 scope.go:117] "RemoveContainer" containerID="deafec397eec2d6c23d4c77c4498feab526dfe9f7c5457ee1e934a2c553e6e0e"
Sep 17 16:19:35.913226 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:35.913205    2580 scope.go:117] "RemoveContainer" containerID="0892a6aa6eb3dd4d4ac462d329eba26d4e0b812ee0cc02a9022ddafb064f2b50"
Sep 17 16:19:42.793435 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:42.793394    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:19:44.806269 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:44.806235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:52.793281 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:19:52.793242    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:20:02.793133 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:02.793093    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:20:12.793618 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:12.793577    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:20:22.793586 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:22.793539    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:20:32.793374 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:32.793278    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:20:42.793772 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:42.793739    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:20:49.561141 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:49.561103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:50.828619 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:50.828586    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"]
Sep 17 16:20:50.829080 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:50.828834    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" containerID="cri-o://8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4" gracePeriod=30
Sep 17 16:20:50.829080 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:50.828918    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kube-rbac-proxy" containerID="cri-o://0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360" gracePeriod=30
Sep 17 16:20:50.858653 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:50.858618    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:51.110017 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:51.109983    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerID="0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360" exitCode=2
Sep 17 16:20:51.110184 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:51.110058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" event={"ID":"7ac5406e-d16c-43b0-bf52-6a82d71d294f","Type":"ContainerDied","Data":"0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360"}
Sep 17 16:20:52.788024 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:52.787984    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 16:20:52.793006 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:52.792972    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 16:20:55.613668 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.613639    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:20:55.718454 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.718364    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbcbm\" (UniqueName: \"kubernetes.io/projected/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-api-access-gbcbm\") pod \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") "
Sep 17 16:20:55.718454 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.718424    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac5406e-d16c-43b0-bf52-6a82d71d294f-proxy-tls\") pod \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") "
Sep 17 16:20:55.718651 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.718463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-rbac-proxy-sar-config\") pod \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") "
Sep 17 16:20:55.718651 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.718560    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-cabundle-cert\") pod \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") "
Sep 17 16:20:55.718651 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.718591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kserve-provision-location\") pod \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\" (UID: \"7ac5406e-d16c-43b0-bf52-6a82d71d294f\") "
Sep 17 16:20:55.718950 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.718931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ac5406e-d16c-43b0-bf52-6a82d71d294f" (UID: "7ac5406e-d16c-43b0-bf52-6a82d71d294f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:20:55.719046 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.718952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ac5406e-d16c-43b0-bf52-6a82d71d294f" (UID: "7ac5406e-d16c-43b0-bf52-6a82d71d294f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:20:55.719046 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.719006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ac5406e-d16c-43b0-bf52-6a82d71d294f" (UID: "7ac5406e-d16c-43b0-bf52-6a82d71d294f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:20:55.720889 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.720857    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-api-access-gbcbm" (OuterVolumeSpecName: "kube-api-access-gbcbm") pod "7ac5406e-d16c-43b0-bf52-6a82d71d294f" (UID: "7ac5406e-d16c-43b0-bf52-6a82d71d294f"). InnerVolumeSpecName "kube-api-access-gbcbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:20:55.720889 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.720887    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac5406e-d16c-43b0-bf52-6a82d71d294f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ac5406e-d16c-43b0-bf52-6a82d71d294f" (UID: "7ac5406e-d16c-43b0-bf52-6a82d71d294f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:20:55.819658 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.819623    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:20:55.819658 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.819654    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ac5406e-d16c-43b0-bf52-6a82d71d294f-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:20:55.819658 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.819663    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:20:55.819929 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.819672    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbcbm\" (UniqueName: \"kubernetes.io/projected/7ac5406e-d16c-43b0-bf52-6a82d71d294f-kube-api-access-gbcbm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:20:55.819929 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.819680    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ac5406e-d16c-43b0-bf52-6a82d71d294f-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:20:55.974410 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:55.974334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:56.136515 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.136476    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerID="8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4" exitCode=0
Sep 17 16:20:56.136696 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.136537    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"
Sep 17 16:20:56.136696 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.136554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" event={"ID":"7ac5406e-d16c-43b0-bf52-6a82d71d294f","Type":"ContainerDied","Data":"8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4"}
Sep 17 16:20:56.136696 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.136613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4" event={"ID":"7ac5406e-d16c-43b0-bf52-6a82d71d294f","Type":"ContainerDied","Data":"dfa743af3b89d2b47e8d09bcfdda4b4e6560b8d962e88bcf8f8a194103192948"}
Sep 17 16:20:56.136696 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.136640    2580 scope.go:117] "RemoveContainer" containerID="0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360"
Sep 17 16:20:56.146180 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.146158    2580 scope.go:117] "RemoveContainer" containerID="8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4"
Sep 17 16:20:56.154611 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.154579    2580 scope.go:117] "RemoveContainer" containerID="bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f"
Sep 17 16:20:56.158651 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.158622    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"]
Sep 17 16:20:56.163077 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.163056    2580 scope.go:117] "RemoveContainer" containerID="0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360"
Sep 17 16:20:56.163436 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:20:56.163406    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360\": container with ID starting with 0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360 not found: ID does not exist" containerID="0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360"
Sep 17 16:20:56.163489 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.163446    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360"} err="failed to get container status \"0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360\": rpc error: code = NotFound desc = could not find container \"0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360\": container with ID starting with 0f44c8c9582cc5a3ffb02f3aa567acd2e2a9bbe068f2be5069fc2917cc544360 not found: ID does not exist"
Sep 17 16:20:56.163489 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.163466    2580 scope.go:117] "RemoveContainer" containerID="8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4"
Sep 17 16:20:56.163746 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:20:56.163733    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4\": container with ID starting with 8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4 not found: ID does not exist" containerID="8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4"
Sep 17 16:20:56.163782 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.163750    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4"} err="failed to get container status \"8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4\": rpc error: code = NotFound desc = could not find container \"8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4\": container with ID starting with 8fd0d5d4ebcbfb7ea0b1ae1671d84e1f603d8799923f3fc17c245d2501148ff4 not found: ID does not exist"
Sep 17 16:20:56.163782 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.163763    2580 scope.go:117] "RemoveContainer" containerID="bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f"
Sep 17 16:20:56.164020 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:20:56.164003    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f\": container with ID starting with bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f not found: ID does not exist" containerID="bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f"
Sep 17 16:20:56.164062 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.164026    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f"} err="failed to get container status \"bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f\": rpc error: code = NotFound desc = could not find container \"bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f\": container with ID starting with bce119187ad8e7f33bf645d7f61e8502c6427b1211b9ef37ae1e04b814e9fa2f not found: ID does not exist"
Sep 17 16:20:56.168252 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.168225    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tl8h4"]
Sep 17 16:20:56.174954 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:56.174923    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:57.674844 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:20:57.674794    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" path="/var/lib/kubelet/pods/7ac5406e-d16c-43b0-bf52-6a82d71d294f/volumes"
Sep 17 16:21:00.109852 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:21:00.109791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:05.189353 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:21:05.189320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:05.342034 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:21:05.341992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:10.502668 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:21:10.502626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:55.321499 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:21:55.321457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:00.572585 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"]
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572857    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572871    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572881    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572886    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572893    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kube-rbac-proxy"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572898    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kube-rbac-proxy"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572905    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="storage-initializer"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572910    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="storage-initializer"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572916    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kube-rbac-proxy"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572924    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kube-rbac-proxy"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572939    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="storage-initializer"
Sep 17 16:22:00.572977 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572944    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="storage-initializer"
Sep 17 16:22:00.573343 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.572991    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kserve-container"
Sep 17 16:22:00.573343 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.573000    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kserve-container"
Sep 17 16:22:00.573343 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.573005    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c90e3f2-0bfa-459b-8c96-e8ac9ddc66f4" containerName="kube-rbac-proxy"
Sep 17 16:22:00.573343 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.573011    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac5406e-d16c-43b0-bf52-6a82d71d294f" containerName="kube-rbac-proxy"
Sep 17 16:22:00.576037 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.576019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.580188 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.580159    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:22:00.580351 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.580202    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 16:22:00.580351 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.580161    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:22:00.583367 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.583343    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:22:00.583504 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.583369    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 16:22:00.583870 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.583854    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:22:00.583965 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.583892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:22:00.595461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.595435    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"]
Sep 17 16:22:00.629375 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.629336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ab393c-243f-46de-bf53-6b42943db279-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.629542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.629381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ab393c-243f-46de-bf53-6b42943db279-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.629542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.629481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.629542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.629510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.629542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.629531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44q4l\" (UniqueName: \"kubernetes.io/projected/e7ab393c-243f-46de-bf53-6b42943db279-kube-api-access-44q4l\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.695493 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.695432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:00.730295 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.730256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.730461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.730302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.730461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.730329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44q4l\" (UniqueName: \"kubernetes.io/projected/e7ab393c-243f-46de-bf53-6b42943db279-kube-api-access-44q4l\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.730461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.730385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ab393c-243f-46de-bf53-6b42943db279-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.730461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.730418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ab393c-243f-46de-bf53-6b42943db279-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.730866 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.730846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ab393c-243f-46de-bf53-6b42943db279-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.731078 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.731054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.731366 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.731059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.733025 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.733005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ab393c-243f-46de-bf53-6b42943db279-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.739951 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.739930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44q4l\" (UniqueName: \"kubernetes.io/projected/e7ab393c-243f-46de-bf53-6b42943db279-kube-api-access-44q4l\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:00.885757 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:00.885728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:01.190510 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:01.190487    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"]
Sep 17 16:22:01.192851 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:22:01.192815    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ab393c_243f_46de_bf53_6b42943db279.slice/crio-1a73c60566335b7dee86de6778bc93749ec684088b28681496e987a964f2cd19 WatchSource:0}: Error finding container 1a73c60566335b7dee86de6778bc93749ec684088b28681496e987a964f2cd19: Status 404 returned error can't find the container with id 1a73c60566335b7dee86de6778bc93749ec684088b28681496e987a964f2cd19
Sep 17 16:22:01.195987 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:01.195960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:01.380210 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:01.380170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" event={"ID":"e7ab393c-243f-46de-bf53-6b42943db279","Type":"ContainerStarted","Data":"464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91"}
Sep 17 16:22:01.380389 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:01.380261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" event={"ID":"e7ab393c-243f-46de-bf53-6b42943db279","Type":"ContainerStarted","Data":"1a73c60566335b7dee86de6778bc93749ec684088b28681496e987a964f2cd19"}
Sep 17 16:22:01.639115 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:01.639081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:05.397303 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:05.397267    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7ab393c-243f-46de-bf53-6b42943db279" containerID="464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91" exitCode=0
Sep 17 16:22:05.397689 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:05.397322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" event={"ID":"e7ab393c-243f-46de-bf53-6b42943db279","Type":"ContainerDied","Data":"464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91"}
Sep 17 16:22:06.403583 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:06.403542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" event={"ID":"e7ab393c-243f-46de-bf53-6b42943db279","Type":"ContainerStarted","Data":"d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78"}
Sep 17 16:22:06.404080 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:06.403634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" event={"ID":"e7ab393c-243f-46de-bf53-6b42943db279","Type":"ContainerStarted","Data":"aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59"}
Sep 17 16:22:06.404080 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:06.403982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:06.404227 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:06.404123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:06.405617 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:06.405590    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 16:22:06.423445 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:06.423379    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" podStartSLOduration=2.370275 podStartE2EDuration="6.423361642s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.053086642s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:22:06.421762413 +0000 UTC m=+2611.264125219" watchObservedRunningTime="2026-09-17 16:22:06.423361642 +0000 UTC m=+2611.265724446"
Sep 17 16:22:07.409159 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:07.409116    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 16:22:12.413528 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:12.413498    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:12.414219 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:12.414188    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 16:22:22.415598 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:22.415561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:30.605233 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:30.605196    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"]
Sep 17 16:22:30.605764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:30.605447    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kserve-container" containerID="cri-o://aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59" gracePeriod=30
Sep 17 16:22:30.605764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:30.605499    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kube-rbac-proxy" containerID="cri-o://d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78" gracePeriod=30
Sep 17 16:22:30.639334 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:30.639305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:31.443015 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.442989    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:31.505957 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.505852    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7ab393c-243f-46de-bf53-6b42943db279" containerID="d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78" exitCode=2
Sep 17 16:22:31.505957 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.505880    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7ab393c-243f-46de-bf53-6b42943db279" containerID="aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59" exitCode=0
Sep 17 16:22:31.505957 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.505903    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"
Sep 17 16:22:31.505957 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.505915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" event={"ID":"e7ab393c-243f-46de-bf53-6b42943db279","Type":"ContainerDied","Data":"d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78"}
Sep 17 16:22:31.506261 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.505978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" event={"ID":"e7ab393c-243f-46de-bf53-6b42943db279","Type":"ContainerDied","Data":"aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59"}
Sep 17 16:22:31.506261 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.506006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx" event={"ID":"e7ab393c-243f-46de-bf53-6b42943db279","Type":"ContainerDied","Data":"1a73c60566335b7dee86de6778bc93749ec684088b28681496e987a964f2cd19"}
Sep 17 16:22:31.506261 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.506039    2580 scope.go:117] "RemoveContainer" containerID="d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78"
Sep 17 16:22:31.514259 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.514241    2580 scope.go:117] "RemoveContainer" containerID="aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59"
Sep 17 16:22:31.521658 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.521638    2580 scope.go:117] "RemoveContainer" containerID="464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91"
Sep 17 16:22:31.528853 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.528832    2580 scope.go:117] "RemoveContainer" containerID="d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78"
Sep 17 16:22:31.529140 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:22:31.529121    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78\": container with ID starting with d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78 not found: ID does not exist" containerID="d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78"
Sep 17 16:22:31.529201 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.529150    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78"} err="failed to get container status \"d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78\": rpc error: code = NotFound desc = could not find container \"d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78\": container with ID starting with d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78 not found: ID does not exist"
Sep 17 16:22:31.529201 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.529169    2580 scope.go:117] "RemoveContainer" containerID="aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59"
Sep 17 16:22:31.529418 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:22:31.529404    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59\": container with ID starting with aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59 not found: ID does not exist" containerID="aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59"
Sep 17 16:22:31.529460 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.529422    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59"} err="failed to get container status \"aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59\": rpc error: code = NotFound desc = could not find container \"aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59\": container with ID starting with aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59 not found: ID does not exist"
Sep 17 16:22:31.529460 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.529434    2580 scope.go:117] "RemoveContainer" containerID="464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91"
Sep 17 16:22:31.529694 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:22:31.529676    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91\": container with ID starting with 464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91 not found: ID does not exist" containerID="464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91"
Sep 17 16:22:31.529745 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.529698    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91"} err="failed to get container status \"464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91\": rpc error: code = NotFound desc = could not find container \"464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91\": container with ID starting with 464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91 not found: ID does not exist"
Sep 17 16:22:31.529745 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.529714    2580 scope.go:117] "RemoveContainer" containerID="d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78"
Sep 17 16:22:31.529998 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.529978    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78"} err="failed to get container status \"d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78\": rpc error: code = NotFound desc = could not find container \"d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78\": container with ID starting with d336c3456e64f9c93d5007fc9636dc593ce3fe95fb55bc91ce3cd08f6b773c78 not found: ID does not exist"
Sep 17 16:22:31.529998 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.529997    2580 scope.go:117] "RemoveContainer" containerID="aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59"
Sep 17 16:22:31.530240 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.530223    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59"} err="failed to get container status \"aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59\": rpc error: code = NotFound desc = could not find container \"aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59\": container with ID starting with aa1dc3c5e687cb451956a7679edf0f15b78de6213a9f12537bd123e8b9de8c59 not found: ID does not exist"
Sep 17 16:22:31.530285 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.530241    2580 scope.go:117] "RemoveContainer" containerID="464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91"
Sep 17 16:22:31.530470 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.530454    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91"} err="failed to get container status \"464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91\": rpc error: code = NotFound desc = could not find container \"464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91\": container with ID starting with 464944ba63022df1804662aae3eb04de620df780780bc79a9a21042d62deac91 not found: ID does not exist"
Sep 17 16:22:31.591946 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.591907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ab393c-243f-46de-bf53-6b42943db279-proxy-tls\") pod \"e7ab393c-243f-46de-bf53-6b42943db279\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") "
Sep 17 16:22:31.592144 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.591965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44q4l\" (UniqueName: \"kubernetes.io/projected/e7ab393c-243f-46de-bf53-6b42943db279-kube-api-access-44q4l\") pod \"e7ab393c-243f-46de-bf53-6b42943db279\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") "
Sep 17 16:22:31.592144 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.592101    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-cabundle-cert\") pod \"e7ab393c-243f-46de-bf53-6b42943db279\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") "
Sep 17 16:22:31.592264 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.592142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ab393c-243f-46de-bf53-6b42943db279-kserve-provision-location\") pod \"e7ab393c-243f-46de-bf53-6b42943db279\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") "
Sep 17 16:22:31.592264 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.592189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-kube-rbac-proxy-sar-config\") pod \"e7ab393c-243f-46de-bf53-6b42943db279\" (UID: \"e7ab393c-243f-46de-bf53-6b42943db279\") "
Sep 17 16:22:31.592482 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.592459    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7ab393c-243f-46de-bf53-6b42943db279" (UID: "e7ab393c-243f-46de-bf53-6b42943db279"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:22:31.592618 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.592524    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ab393c-243f-46de-bf53-6b42943db279-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7ab393c-243f-46de-bf53-6b42943db279" (UID: "e7ab393c-243f-46de-bf53-6b42943db279"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:22:31.592618 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.592547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7ab393c-243f-46de-bf53-6b42943db279" (UID: "e7ab393c-243f-46de-bf53-6b42943db279"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:22:31.594313 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.594289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ab393c-243f-46de-bf53-6b42943db279-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7ab393c-243f-46de-bf53-6b42943db279" (UID: "e7ab393c-243f-46de-bf53-6b42943db279"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:22:31.594415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.594385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ab393c-243f-46de-bf53-6b42943db279-kube-api-access-44q4l" (OuterVolumeSpecName: "kube-api-access-44q4l") pod "e7ab393c-243f-46de-bf53-6b42943db279" (UID: "e7ab393c-243f-46de-bf53-6b42943db279"). InnerVolumeSpecName "kube-api-access-44q4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:22:31.693449 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.693417    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44q4l\" (UniqueName: \"kubernetes.io/projected/e7ab393c-243f-46de-bf53-6b42943db279-kube-api-access-44q4l\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:22:31.693449 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.693447    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:22:31.693850 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.693456    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7ab393c-243f-46de-bf53-6b42943db279-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:22:31.693850 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.693465    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7ab393c-243f-46de-bf53-6b42943db279-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:22:31.693850 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.693474    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7ab393c-243f-46de-bf53-6b42943db279-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:22:31.825486 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.825396    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"]
Sep 17 16:22:31.829465 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:31.829440    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mtjkx"]
Sep 17 16:22:33.674463 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:33.674423    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7ab393c-243f-46de-bf53-6b42943db279" path="/var/lib/kubelet/pods/e7ab393c-243f-46de-bf53-6b42943db279/volumes"
Sep 17 16:22:35.775150 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:35.775119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:36.322060 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:22:36.321975    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:21.901361 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:21.901324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:25.955051 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:25.955000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:27.113048 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.112960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:27.148656 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.148613    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"]
Sep 17 16:23:27.148979 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.148964    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kserve-container"
Sep 17 16:23:27.149031 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.148981    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kserve-container"
Sep 17 16:23:27.149031 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.148996    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="storage-initializer"
Sep 17 16:23:27.149031 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.149004    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="storage-initializer"
Sep 17 16:23:27.149031 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.149015    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kube-rbac-proxy"
Sep 17 16:23:27.149031 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.149023    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kube-rbac-proxy"
Sep 17 16:23:27.149191 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.149070    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kserve-container"
Sep 17 16:23:27.149191 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.149079    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ab393c-243f-46de-bf53-6b42943db279" containerName="kube-rbac-proxy"
Sep 17 16:23:27.152128 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.152108    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.154288 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.154262    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 16:23:27.154502 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.154264    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 16:23:27.154502 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.154497    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:23:27.154611 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.154570    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 16:23:27.154648 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.154624    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:23:27.154908 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.154890    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:23:27.167008 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.166973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"]
Sep 17 16:23:27.212560 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.212512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vwvk\" (UniqueName: \"kubernetes.io/projected/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-api-access-7vwvk\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.212560 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.212557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.212822 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.212642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.212822 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.212724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-proxy-tls\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.233463 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.233428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:27.314044 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.313999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vwvk\" (UniqueName: \"kubernetes.io/projected/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-api-access-7vwvk\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.314044 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.314042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.314332 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.314231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.314332 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.314302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-proxy-tls\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.314467 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.314445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.314894 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.314868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.317324 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.317301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-proxy-tls\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.323795 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.323765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vwvk\" (UniqueName: \"kubernetes.io/projected/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-api-access-7vwvk\") pod \"isvc-sklearn-s3-predictor-547bfff996-2r8wd\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.463137 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.463035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:27.741716 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:27.741687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"]
Sep 17 16:23:27.744019 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:23:27.743989    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d83f66e_fee8_42b8_b9a6_51550d4ee5be.slice/crio-1389813fb1eb0ab943ce96f16b5ec0fcb39a10355f68d52e8bfe739dcbd0940e WatchSource:0}: Error finding container 1389813fb1eb0ab943ce96f16b5ec0fcb39a10355f68d52e8bfe739dcbd0940e: Status 404 returned error can't find the container with id 1389813fb1eb0ab943ce96f16b5ec0fcb39a10355f68d52e8bfe739dcbd0940e
Sep 17 16:23:28.721055 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:28.721012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" event={"ID":"5d83f66e-fee8-42b8-b9a6-51550d4ee5be","Type":"ContainerStarted","Data":"f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa"}
Sep 17 16:23:28.721536 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:28.721086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" event={"ID":"5d83f66e-fee8-42b8-b9a6-51550d4ee5be","Type":"ContainerStarted","Data":"1389813fb1eb0ab943ce96f16b5ec0fcb39a10355f68d52e8bfe739dcbd0940e"}
Sep 17 16:23:29.726254 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:29.726215    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerID="f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa" exitCode=0
Sep 17 16:23:29.726745 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:29.726300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" event={"ID":"5d83f66e-fee8-42b8-b9a6-51550d4ee5be","Type":"ContainerDied","Data":"f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa"}
Sep 17 16:23:30.732876 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:30.732834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" event={"ID":"5d83f66e-fee8-42b8-b9a6-51550d4ee5be","Type":"ContainerStarted","Data":"670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e"}
Sep 17 16:23:30.733383 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:30.732902    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" event={"ID":"5d83f66e-fee8-42b8-b9a6-51550d4ee5be","Type":"ContainerStarted","Data":"72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e"}
Sep 17 16:23:30.733383 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:30.732986    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:30.756524 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:30.756466    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podStartSLOduration=2.3092792810000002 podStartE2EDuration="3.75644911s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.447169829s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:23:30.755180193 +0000 UTC m=+2695.597542997" watchObservedRunningTime="2026-09-17 16:23:30.75644911 +0000 UTC m=+2695.598811915"
Sep 17 16:23:31.070724 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.070631    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:31.103572 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.103537    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"]
Sep 17 16:23:31.107280 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.107257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.109346 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.109315    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 16:23:31.109346 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.109328    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:23:31.109560 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.109329    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:23:31.109560 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.109451    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 16:23:31.121035 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.121005    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"]
Sep 17 16:23:31.170209 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.170172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:31.250585 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.250536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a285c4-2d1a-4539-a6de-36632617d01c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.250762 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.250593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwx9\" (UniqueName: \"kubernetes.io/projected/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-api-access-ngwx9\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.250762 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.250678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.250898 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.250759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8a285c4-2d1a-4539-a6de-36632617d01c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.250898 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.250855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.351867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.351733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.351867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.351812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8a285c4-2d1a-4539-a6de-36632617d01c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.351867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.351847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.351867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.351869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a285c4-2d1a-4539-a6de-36632617d01c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.352191 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.351896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwx9\" (UniqueName: \"kubernetes.io/projected/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-api-access-ngwx9\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.352282 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.352264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a285c4-2d1a-4539-a6de-36632617d01c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.352518 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.352500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.352572 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.352532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.354539 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.354517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8a285c4-2d1a-4539-a6de-36632617d01c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.362590 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.362554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwx9\" (UniqueName: \"kubernetes.io/projected/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-api-access-ngwx9\") pod \"isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.417578 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.417532    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:31.702960 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.702929    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"]
Sep 17 16:23:31.705569 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:23:31.705539    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a285c4_2d1a_4539_a6de_36632617d01c.slice/crio-8cca18fe5b250237660024cee0f2fd6127d0abf95bae94067c4b5d87c56235de WatchSource:0}: Error finding container 8cca18fe5b250237660024cee0f2fd6127d0abf95bae94067c4b5d87c56235de: Status 404 returned error can't find the container with id 8cca18fe5b250237660024cee0f2fd6127d0abf95bae94067c4b5d87c56235de
Sep 17 16:23:31.737926 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.737890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerStarted","Data":"8cca18fe5b250237660024cee0f2fd6127d0abf95bae94067c4b5d87c56235de"}
Sep 17 16:23:31.738409 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.738221    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:31.739496 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:31.739471    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:23:32.743868 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:32.743824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerStarted","Data":"332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e"}
Sep 17 16:23:32.744327 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:32.744152    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:23:35.795957 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:35.795925    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:23:35.802609 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:35.802584    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:23:36.761085 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:36.761048    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerID="332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e" exitCode=0
Sep 17 16:23:36.761246 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:36.761119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerDied","Data":"332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e"}
Sep 17 16:23:37.750371 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:37.750334    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:23:37.751106 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:37.751076    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:23:37.772774 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:37.772720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerStarted","Data":"323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d"}
Sep 17 16:23:42.801267 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:42.801226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerStarted","Data":"fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848"}
Sep 17 16:23:42.801777 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:42.801304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerStarted","Data":"07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece"}
Sep 17 16:23:42.801777 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:42.801436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:42.801777 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:42.801503    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:42.801777 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:42.801527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:42.823998 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:42.823921    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podStartSLOduration=2.470652121 podStartE2EDuration="11.823898751s" totalImagesPullingTime="5.183346622s" totalInitContainerRuntime="4.169900008s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:23:42.822646368 +0000 UTC m=+2707.665009197" watchObservedRunningTime="2026-09-17 16:23:42.823898751 +0000 UTC m=+2707.666261557"
Sep 17 16:23:47.751872 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:47.751826    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:23:48.121834 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:48.121766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:48.811005 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:48.810968    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:23:57.751497 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:23:57.751453    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:24:07.751114 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:07.751069    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:24:08.812419 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:08.812370    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 16:24:11.418218 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:11.418171    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 16:24:17.751952 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:17.751912    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:24:18.814750 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:18.814712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:24:27.751010 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:27.750959    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:24:37.750955 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:37.750912    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:24:47.751549 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:47.751516    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:24:48.816484 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:48.816453    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:24:51.269382 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:51.269349    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"]
Sep 17 16:24:51.269790 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:51.269619    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy" containerID="cri-o://fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848" gracePeriod=30
Sep 17 16:24:51.269790 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:51.269634    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-agent" containerID="cri-o://07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece" gracePeriod=30
Sep 17 16:24:51.269790 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:51.269598    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-container" containerID="cri-o://323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d" gracePeriod=30
Sep 17 16:24:51.320999 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:51.320968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:52.087974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:52.087942    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerID="fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848" exitCode=2
Sep 17 16:24:52.088159 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:52.088014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerDied","Data":"fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848"}
Sep 17 16:24:53.805610 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:53.805566    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 16:24:54.099255 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:54.099159    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerID="323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d" exitCode=0
Sep 17 16:24:54.099255 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:54.099231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerDied","Data":"323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d"}
Sep 17 16:24:56.335600 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:56.335558    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:56.621215 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:56.621183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:57.252142 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:57.252100    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"]
Sep 17 16:24:57.252426 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:57.252383    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" containerID="cri-o://72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e" gracePeriod=30
Sep 17 16:24:57.252530 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:57.252463    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kube-rbac-proxy" containerID="cri-o://670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e" gracePeriod=30
Sep 17 16:24:57.259304 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:57.259277    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:57.447589 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:57.447544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:57.744499 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:57.744461    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 17 16:24:57.751755 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:57.751716    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 16:24:57.967470 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:57.967425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:58.118309 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:58.118269    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerID="670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e" exitCode=2
Sep 17 16:24:58.118500 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:58.118348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" event={"ID":"5d83f66e-fee8-42b8-b9a6-51550d4ee5be","Type":"ContainerDied","Data":"670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e"}
Sep 17 16:24:58.806433 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:58.806387    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 16:24:58.811843 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:24:58.811788    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 16:25:02.295142 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.295115    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:25:02.413419 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.413387    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kserve-provision-location\") pod \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") "
Sep 17 16:25:02.413419 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.413422    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-rbac-proxy-sar-config\") pod \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") "
Sep 17 16:25:02.413696 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.413445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-proxy-tls\") pod \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") "
Sep 17 16:25:02.413696 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.413527    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vwvk\" (UniqueName: \"kubernetes.io/projected/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-api-access-7vwvk\") pod \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\" (UID: \"5d83f66e-fee8-42b8-b9a6-51550d4ee5be\") "
Sep 17 16:25:02.413840 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.413691    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d83f66e-fee8-42b8-b9a6-51550d4ee5be" (UID: "5d83f66e-fee8-42b8-b9a6-51550d4ee5be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:02.413840 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.413785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d83f66e-fee8-42b8-b9a6-51550d4ee5be" (UID: "5d83f66e-fee8-42b8-b9a6-51550d4ee5be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:02.415731 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.415709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-api-access-7vwvk" (OuterVolumeSpecName: "kube-api-access-7vwvk") pod "5d83f66e-fee8-42b8-b9a6-51550d4ee5be" (UID: "5d83f66e-fee8-42b8-b9a6-51550d4ee5be"). InnerVolumeSpecName "kube-api-access-7vwvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:25:02.415794 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.415745    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d83f66e-fee8-42b8-b9a6-51550d4ee5be" (UID: "5d83f66e-fee8-42b8-b9a6-51550d4ee5be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:25:02.514348 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.514312    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:25:02.514348 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.514341    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:25:02.514348 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.514353    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:25:02.514587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:02.514362    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vwvk\" (UniqueName: \"kubernetes.io/projected/5d83f66e-fee8-42b8-b9a6-51550d4ee5be-kube-api-access-7vwvk\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:25:03.139908 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.139869    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerID="72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e" exitCode=0
Sep 17 16:25:03.140090 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.139933    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"
Sep 17 16:25:03.140090 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.139957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" event={"ID":"5d83f66e-fee8-42b8-b9a6-51550d4ee5be","Type":"ContainerDied","Data":"72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e"}
Sep 17 16:25:03.140090 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.140012    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd" event={"ID":"5d83f66e-fee8-42b8-b9a6-51550d4ee5be","Type":"ContainerDied","Data":"1389813fb1eb0ab943ce96f16b5ec0fcb39a10355f68d52e8bfe739dcbd0940e"}
Sep 17 16:25:03.140090 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.140037    2580 scope.go:117] "RemoveContainer" containerID="670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e"
Sep 17 16:25:03.148920 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.148899    2580 scope.go:117] "RemoveContainer" containerID="72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e"
Sep 17 16:25:03.157180 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.157162    2580 scope.go:117] "RemoveContainer" containerID="f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa"
Sep 17 16:25:03.165542 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.165329    2580 scope.go:117] "RemoveContainer" containerID="670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e"
Sep 17 16:25:03.165658 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:25:03.165633    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e\": container with ID starting with 670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e not found: ID does not exist" containerID="670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e"
Sep 17 16:25:03.165707 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.165671    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e"} err="failed to get container status \"670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e\": rpc error: code = NotFound desc = could not find container \"670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e\": container with ID starting with 670d8003ea1d13790144a93513b85ba1ae0e0c6feb54655690e3f0f4d17f146e not found: ID does not exist"
Sep 17 16:25:03.165707 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.165696    2580 scope.go:117] "RemoveContainer" containerID="72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e"
Sep 17 16:25:03.166000 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:25:03.165983    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e\": container with ID starting with 72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e not found: ID does not exist" containerID="72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e"
Sep 17 16:25:03.166067 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.166006    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e"} err="failed to get container status \"72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e\": rpc error: code = NotFound desc = could not find container \"72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e\": container with ID starting with 72b31411ddb9994dd9e67007cb7617f033e91c02f4cb5a2dc53dec3aa035b98e not found: ID does not exist"
Sep 17 16:25:03.166067 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.166021    2580 scope.go:117] "RemoveContainer" containerID="f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa"
Sep 17 16:25:03.166183 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.166167    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"]
Sep 17 16:25:03.166277 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:25:03.166260    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa\": container with ID starting with f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa not found: ID does not exist" containerID="f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa"
Sep 17 16:25:03.166317 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.166286    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa"} err="failed to get container status \"f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa\": rpc error: code = NotFound desc = could not find container \"f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa\": container with ID starting with f66c82451395b6a2afc0cd1adaa7dde6150be9bbfb22e66a0a8799d7743e68fa not found: ID does not exist"
Sep 17 16:25:03.172110 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.172090    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-547bfff996-2r8wd"]
Sep 17 16:25:03.678817 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.676312    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" path="/var/lib/kubelet/pods/5d83f66e-fee8-42b8-b9a6-51550d4ee5be/volumes"
Sep 17 16:25:03.805668 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.805618    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 16:25:03.805909 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:03.805762    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:25:08.806140 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:08.806097    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 16:25:08.812636 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:08.812598    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 16:25:13.805745 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:13.805694    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 16:25:18.805734 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:18.805683    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 16:25:18.812269 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:18.812236    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 16:25:18.812393 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:18.812355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:25:21.427708 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.427682    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:25:21.451786 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.451751    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-rbac-proxy-sar-config\") pod \"f8a285c4-2d1a-4539-a6de-36632617d01c\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") "
Sep 17 16:25:21.451786 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.451785    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-cabundle-cert\") pod \"f8a285c4-2d1a-4539-a6de-36632617d01c\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") "
Sep 17 16:25:21.452023 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.451825    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngwx9\" (UniqueName: \"kubernetes.io/projected/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-api-access-ngwx9\") pod \"f8a285c4-2d1a-4539-a6de-36632617d01c\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") "
Sep 17 16:25:21.452023 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.451856    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8a285c4-2d1a-4539-a6de-36632617d01c-proxy-tls\") pod \"f8a285c4-2d1a-4539-a6de-36632617d01c\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") "
Sep 17 16:25:21.452023 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.451912    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a285c4-2d1a-4539-a6de-36632617d01c-kserve-provision-location\") pod \"f8a285c4-2d1a-4539-a6de-36632617d01c\" (UID: \"f8a285c4-2d1a-4539-a6de-36632617d01c\") "
Sep 17 16:25:21.452275 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.452237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a285c4-2d1a-4539-a6de-36632617d01c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8a285c4-2d1a-4539-a6de-36632617d01c" (UID: "f8a285c4-2d1a-4539-a6de-36632617d01c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:21.452275 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.452252    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f8a285c4-2d1a-4539-a6de-36632617d01c" (UID: "f8a285c4-2d1a-4539-a6de-36632617d01c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:21.452403 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.452273    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8a285c4-2d1a-4539-a6de-36632617d01c" (UID: "f8a285c4-2d1a-4539-a6de-36632617d01c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:21.454127 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.454107    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a285c4-2d1a-4539-a6de-36632617d01c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8a285c4-2d1a-4539-a6de-36632617d01c" (UID: "f8a285c4-2d1a-4539-a6de-36632617d01c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:25:21.454298 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.454276    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-api-access-ngwx9" (OuterVolumeSpecName: "kube-api-access-ngwx9") pod "f8a285c4-2d1a-4539-a6de-36632617d01c" (UID: "f8a285c4-2d1a-4539-a6de-36632617d01c"). InnerVolumeSpecName "kube-api-access-ngwx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:25:21.552770 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.552665    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:25:21.552770 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.552701    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8a285c4-2d1a-4539-a6de-36632617d01c-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:25:21.552770 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.552714    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngwx9\" (UniqueName: \"kubernetes.io/projected/f8a285c4-2d1a-4539-a6de-36632617d01c-kube-api-access-ngwx9\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:25:21.552770 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.552726    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8a285c4-2d1a-4539-a6de-36632617d01c-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:25:21.552770 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:21.552737    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8a285c4-2d1a-4539-a6de-36632617d01c-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:25:22.216716 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.216682    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerID="07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece" exitCode=0
Sep 17 16:25:22.216903 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.216744    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"
Sep 17 16:25:22.216903 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.216772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerDied","Data":"07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece"}
Sep 17 16:25:22.216903 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.216861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj" event={"ID":"f8a285c4-2d1a-4539-a6de-36632617d01c","Type":"ContainerDied","Data":"8cca18fe5b250237660024cee0f2fd6127d0abf95bae94067c4b5d87c56235de"}
Sep 17 16:25:22.216903 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.216895    2580 scope.go:117] "RemoveContainer" containerID="fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848"
Sep 17 16:25:22.225075 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.225063    2580 scope.go:117] "RemoveContainer" containerID="07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece"
Sep 17 16:25:22.232386 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.232360    2580 scope.go:117] "RemoveContainer" containerID="323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d"
Sep 17 16:25:22.238575 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.238549    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"]
Sep 17 16:25:22.240275 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.240261    2580 scope.go:117] "RemoveContainer" containerID="332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e"
Sep 17 16:25:22.245152 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.245133    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c6bf9d8c5-s84qj"]
Sep 17 16:25:22.248265 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.248248    2580 scope.go:117] "RemoveContainer" containerID="fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848"
Sep 17 16:25:22.248540 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:25:22.248521    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848\": container with ID starting with fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848 not found: ID does not exist" containerID="fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848"
Sep 17 16:25:22.248585 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.248554    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848"} err="failed to get container status \"fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848\": rpc error: code = NotFound desc = could not find container \"fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848\": container with ID starting with fb79808ed8147328eb53884cc9e69d1fffd139fe4220bcdbbc42c0595e902848 not found: ID does not exist"
Sep 17 16:25:22.248585 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.248574    2580 scope.go:117] "RemoveContainer" containerID="07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece"
Sep 17 16:25:22.248831 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:25:22.248816    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece\": container with ID starting with 07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece not found: ID does not exist" containerID="07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece"
Sep 17 16:25:22.248886 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.248834    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece"} err="failed to get container status \"07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece\": rpc error: code = NotFound desc = could not find container \"07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece\": container with ID starting with 07f1d53c7227ab2ff82e61ac90b4268f324ffa844d41cbc3c0012a11bc309ece not found: ID does not exist"
Sep 17 16:25:22.248886 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.248849    2580 scope.go:117] "RemoveContainer" containerID="323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d"
Sep 17 16:25:22.249087 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:25:22.249068    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d\": container with ID starting with 323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d not found: ID does not exist" containerID="323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d"
Sep 17 16:25:22.249205 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.249095    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d"} err="failed to get container status \"323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d\": rpc error: code = NotFound desc = could not find container \"323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d\": container with ID starting with 323fa890700276fc34474b34837e3b134c9601f3909b26ae094c3ee3b74d5c9d not found: ID does not exist"
Sep 17 16:25:22.249205 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.249117    2580 scope.go:117] "RemoveContainer" containerID="332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e"
Sep 17 16:25:22.249353 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:25:22.249339    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e\": container with ID starting with 332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e not found: ID does not exist" containerID="332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e"
Sep 17 16:25:22.249395 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:22.249358    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e"} err="failed to get container status \"332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e\": rpc error: code = NotFound desc = could not find container \"332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e\": container with ID starting with 332a81e78518fc0eaff261a4a160854798195914038f7c21d5b7fd1f0e38e47e not found: ID does not exist"
Sep 17 16:25:23.674078 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:23.674044    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" path="/var/lib/kubelet/pods/f8a285c4-2d1a-4539-a6de-36632617d01c/volumes"
Sep 17 16:25:59.013641 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:25:59.013604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:07.787031 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:07.786995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:08.544412 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:08.544379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:12.993396 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:12.993354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:13.043813 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.043760    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"]
Sep 17 16:26:13.044105 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044091    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kube-rbac-proxy"
Sep 17 16:26:13.044164 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044107    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kube-rbac-proxy"
Sep 17 16:26:13.044164 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044116    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-container"
Sep 17 16:26:13.044164 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044122    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-container"
Sep 17 16:26:13.044164 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044127    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy"
Sep 17 16:26:13.044164 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044134    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy"
Sep 17 16:26:13.044164 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044143    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container"
Sep 17 16:26:13.044164 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044149    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container"
Sep 17 16:26:13.044164 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044165    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="storage-initializer"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044171    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="storage-initializer"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044177    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="storage-initializer"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044182    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="storage-initializer"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044189    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-agent"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044194    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-agent"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044239    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-container"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044246    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kserve-agent"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044252    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a285c4-2d1a-4539-a6de-36632617d01c" containerName="kube-rbac-proxy"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044257    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kserve-container"
Sep 17 16:26:13.044415 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.044263    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d83f66e-fee8-42b8-b9a6-51550d4ee5be" containerName="kube-rbac-proxy"
Sep 17 16:26:13.047291 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.047271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.050023 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.050001    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:26:13.050163 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.050120    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:26:13.050230 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.050219    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:26:13.050533 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.050512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:26:13.050647 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.050573    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 16:26:13.050647 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.050593    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:26:13.051227 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.051212    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:26:13.070714 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.070687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"]
Sep 17 16:26:13.163368 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.163322    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.163368 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.163377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.163610 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.163439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.163610 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.163467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.163610 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.163502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4s68\" (UniqueName: \"kubernetes.io/projected/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-api-access-f4s68\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.164161 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.164141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:13.264708 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.264601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.264708 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.264655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.264708 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.264677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.264708 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.264693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.264708 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.264718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4s68\" (UniqueName: \"kubernetes.io/projected/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-api-access-f4s68\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.265200 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.265179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.265338 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.265320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.265444 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.265424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.267492 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.267475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.274916 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.274894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4s68\" (UniqueName: \"kubernetes.io/projected/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-api-access-f4s68\") pod \"isvc-paddle-runtime-predictor-58b99fff76-kt7kv\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.357695 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.357655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:13.638402 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:13.638364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"]
Sep 17 16:26:13.642436 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:26:13.642392    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cab384e_c0f5_4465_9fa8_0ce4be3d33e1.slice/crio-08e9aa4d034a8a79e23535a9275a25e91d757a07cc5e0a8d668f97b2f590345f WatchSource:0}: Error finding container 08e9aa4d034a8a79e23535a9275a25e91d757a07cc5e0a8d668f97b2f590345f: Status 404 returned error can't find the container with id 08e9aa4d034a8a79e23535a9275a25e91d757a07cc5e0a8d668f97b2f590345f
Sep 17 16:26:14.416652 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:14.416614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" event={"ID":"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1","Type":"ContainerStarted","Data":"b5957fddbf70684517df1db094b31cc3bdb3b10bfa87f74109be5d71b1aa1168"}
Sep 17 16:26:14.417051 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:14.416675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" event={"ID":"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1","Type":"ContainerStarted","Data":"08e9aa4d034a8a79e23535a9275a25e91d757a07cc5e0a8d668f97b2f590345f"}
Sep 17 16:26:19.436499 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:19.436463    2580 generic.go:356] "Generic (PLEG): container finished" podID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerID="b5957fddbf70684517df1db094b31cc3bdb3b10bfa87f74109be5d71b1aa1168" exitCode=0
Sep 17 16:26:19.436912 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:19.436537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" event={"ID":"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1","Type":"ContainerDied","Data":"b5957fddbf70684517df1db094b31cc3bdb3b10bfa87f74109be5d71b1aa1168"}
Sep 17 16:26:19.437881 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:19.437867    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:26:27.486399 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:27.486303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:28.550047 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:28.550017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:28.661492 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:28.661448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:31.489869 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:31.489826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" event={"ID":"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1","Type":"ContainerStarted","Data":"70f0d04c763b9ca3b9afbc6d5964a79d580053a2b26893e2a8de447b3c53e419"}
Sep 17 16:26:31.490320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:31.489894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" event={"ID":"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1","Type":"ContainerStarted","Data":"a0044730d6c51fd2f972012d2f6ca2de55de2f97d9e2c73f1d3c3d08de76edec"}
Sep 17 16:26:31.490320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:31.490162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:31.490320 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:31.490294    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:31.491772 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:31.491741    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 16:26:31.518065 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:31.518022    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podStartSLOduration=2.698368818 podStartE2EDuration="18.518007954s" totalImagesPullingTime="11.091334904s" totalInitContainerRuntime="4.728304232s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:26:31.516059893 +0000 UTC m=+2876.358422696" watchObservedRunningTime="2026-09-17 16:26:31.518007954 +0000 UTC m=+2876.360370757"
Sep 17 16:26:32.493732 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:32.493689    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 16:26:37.497875 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:37.497842    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:26:37.498352 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:37.498324    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 16:26:38.635414 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:38.635378    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:39.699466 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.699404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:39.728287 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.728251    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"]
Sep 17 16:26:39.731230 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.731213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.733283 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.733261    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 16:26:39.733425 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.733262    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 16:26:39.733425 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.733264    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 16:26:39.745165 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.745143    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"]
Sep 17 16:26:39.783584 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.783544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.783584 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.783582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.783871 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.783602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.783871 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.783655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.783871 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.783739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tck47\" (UniqueName: \"kubernetes.io/projected/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-api-access-tck47\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.846814 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.846770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:26:39.885058 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.885016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tck47\" (UniqueName: \"kubernetes.io/projected/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-api-access-tck47\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.885238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.885096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.885238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.885123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.885238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.885140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.885238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.885162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.885575 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.885555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.885857 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.885838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.885950 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.885905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.887888 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.887867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:39.898852 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:39.898834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tck47\" (UniqueName: \"kubernetes.io/projected/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-api-access-tck47\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:40.041437 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:40.041342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:40.324936 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:40.324908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"]
Sep 17 16:26:40.327285 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:26:40.327253    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2eefdd_6f9e_4eb1_9574_14ddf2ee232e.slice/crio-7a8e95318c2033bda268b8d901ba973cbd8c80f23d0eb3dac9e01b2fce40d6fc WatchSource:0}: Error finding container 7a8e95318c2033bda268b8d901ba973cbd8c80f23d0eb3dac9e01b2fce40d6fc: Status 404 returned error can't find the container with id 7a8e95318c2033bda268b8d901ba973cbd8c80f23d0eb3dac9e01b2fce40d6fc
Sep 17 16:26:40.526760 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:40.526723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" event={"ID":"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e","Type":"ContainerStarted","Data":"48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf"}
Sep 17 16:26:40.526989 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:40.526786    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" event={"ID":"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e","Type":"ContainerStarted","Data":"7a8e95318c2033bda268b8d901ba973cbd8c80f23d0eb3dac9e01b2fce40d6fc"}
Sep 17 16:26:41.532671 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:41.532643    2580 generic.go:356] "Generic (PLEG): container finished" podID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerID="48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf" exitCode=0
Sep 17 16:26:41.533090 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:41.532692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" event={"ID":"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e","Type":"ContainerDied","Data":"48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf"}
Sep 17 16:26:42.539225 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:42.539176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" event={"ID":"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e","Type":"ContainerStarted","Data":"26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859"}
Sep 17 16:26:42.539645 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:42.539248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" event={"ID":"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e","Type":"ContainerStarted","Data":"8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb"}
Sep 17 16:26:42.539645 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:42.539275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:42.539645 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:42.539304    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:42.540832 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:42.540791    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:26:42.563571 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:42.563518    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podStartSLOduration=2.436875928 podStartE2EDuration="3.563502396s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.126626468s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:26:42.561506432 +0000 UTC m=+2887.403869236" watchObservedRunningTime="2026-09-17 16:26:42.563502396 +0000 UTC m=+2887.405865200"
Sep 17 16:26:43.543621 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:43.543574    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:26:47.499063 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:47.499023    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 16:26:48.548495 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:48.548466    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:26:48.548980 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:48.548954    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:26:57.498776 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:57.498733    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 16:26:58.549481 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:26:58.549442    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:27:07.499253 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:07.499215    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 16:27:08.549707 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:08.549666    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:27:17.498954 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:17.498918    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:27:18.549013 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:18.548977    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:27:24.510278 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:24.510243    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"]
Sep 17 16:27:24.510706 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:24.510462    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container" containerID="cri-o://a0044730d6c51fd2f972012d2f6ca2de55de2f97d9e2c73f1d3c3d08de76edec" gracePeriod=30
Sep 17 16:27:24.510706 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:24.510500    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kube-rbac-proxy" containerID="cri-o://70f0d04c763b9ca3b9afbc6d5964a79d580053a2b26893e2a8de447b3c53e419" gracePeriod=30
Sep 17 16:27:24.552211 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:24.552175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:27:24.711239 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:24.711208    2580 generic.go:356] "Generic (PLEG): container finished" podID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerID="70f0d04c763b9ca3b9afbc6d5964a79d580053a2b26893e2a8de447b3c53e419" exitCode=2
Sep 17 16:27:24.711416 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:24.711289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" event={"ID":"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1","Type":"ContainerDied","Data":"70f0d04c763b9ca3b9afbc6d5964a79d580053a2b26893e2a8de447b3c53e419"}
Sep 17 16:27:27.494603 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.494562    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 17 16:27:27.498988 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.498957    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 16:27:27.725743 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.725706    2580 generic.go:356] "Generic (PLEG): container finished" podID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerID="a0044730d6c51fd2f972012d2f6ca2de55de2f97d9e2c73f1d3c3d08de76edec" exitCode=0
Sep 17 16:27:27.725941 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.725775    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" event={"ID":"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1","Type":"ContainerDied","Data":"a0044730d6c51fd2f972012d2f6ca2de55de2f97d9e2c73f1d3c3d08de76edec"}
Sep 17 16:27:27.755102 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.755042    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:27:27.890993 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.890955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-cabundle-cert\") pod \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") "
Sep 17 16:27:27.891193 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.891018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-rbac-proxy-sar-config\") pod \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") "
Sep 17 16:27:27.891193 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.891037    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-proxy-tls\") pod \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") "
Sep 17 16:27:27.891193 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.891063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kserve-provision-location\") pod \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") "
Sep 17 16:27:27.891193 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.891083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4s68\" (UniqueName: \"kubernetes.io/projected/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-api-access-f4s68\") pod \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\" (UID: \"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1\") "
Sep 17 16:27:27.891413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.891347    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" (UID: "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:27.891413 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.891398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" (UID: "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:27.893284 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.893261    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" (UID: "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:27:27.893394 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.893372    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-api-access-f4s68" (OuterVolumeSpecName: "kube-api-access-f4s68") pod "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" (UID: "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1"). InnerVolumeSpecName "kube-api-access-f4s68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:27:27.905844 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.905786    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" (UID: "8cab384e-c0f5-4465-9fa8-0ce4be3d33e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:27:27.991930 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.991894    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4s68\" (UniqueName: \"kubernetes.io/projected/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-api-access-f4s68\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:27:27.991930 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.991924    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:27:27.991930 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.991933    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:27:27.992165 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.991943    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:27:27.992165 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:27.991951    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:27:28.549754 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:28.549709    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:27:28.732190 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:28.732160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv" event={"ID":"8cab384e-c0f5-4465-9fa8-0ce4be3d33e1","Type":"ContainerDied","Data":"08e9aa4d034a8a79e23535a9275a25e91d757a07cc5e0a8d668f97b2f590345f"}
Sep 17 16:27:28.732360 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:28.732200    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"
Sep 17 16:27:28.732360 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:28.732226    2580 scope.go:117] "RemoveContainer" containerID="70f0d04c763b9ca3b9afbc6d5964a79d580053a2b26893e2a8de447b3c53e419"
Sep 17 16:27:28.741103 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:28.741086    2580 scope.go:117] "RemoveContainer" containerID="a0044730d6c51fd2f972012d2f6ca2de55de2f97d9e2c73f1d3c3d08de76edec"
Sep 17 16:27:28.748892 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:28.748874    2580 scope.go:117] "RemoveContainer" containerID="b5957fddbf70684517df1db094b31cc3bdb3b10bfa87f74109be5d71b1aa1168"
Sep 17 16:27:28.760362 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:28.760336    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"]
Sep 17 16:27:28.765061 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:28.765037    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-kt7kv"]
Sep 17 16:27:29.600945 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:29.600913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:27:29.674747 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:29.674714    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" path="/var/lib/kubelet/pods/8cab384e-c0f5-4465-9fa8-0ce4be3d33e1/volumes"
Sep 17 16:27:29.788642 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:29.788609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:27:38.549323 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:38.549282    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:27:48.549982 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:48.549943    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:27:50.669547 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:27:50.669495    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:28:00.670652 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:00.670622    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:28:09.764464 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:09.764424    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"]
Sep 17 16:28:09.764972 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:09.764664    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" containerID="cri-o://8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb" gracePeriod=30
Sep 17 16:28:09.764972 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:09.764723    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kube-rbac-proxy" containerID="cri-o://26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859" gracePeriod=30
Sep 17 16:28:09.789968 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:09.789937    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:09.890829 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:09.890779    2580 generic.go:356] "Generic (PLEG): container finished" podID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerID="26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859" exitCode=2
Sep 17 16:28:09.890986 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:09.890831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" event={"ID":"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e","Type":"ContainerDied","Data":"26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859"}
Sep 17 16:28:10.669779 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:10.669738    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 16:28:10.868312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:10.868278    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:11.105274 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:11.105185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:13.544313 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:13.544274    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 17 16:28:14.821076 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.821017    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:28:14.911398 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.911363    2580 generic.go:356] "Generic (PLEG): container finished" podID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerID="8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb" exitCode=0
Sep 17 16:28:14.911591 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.911418    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"
Sep 17 16:28:14.911591 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.911439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" event={"ID":"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e","Type":"ContainerDied","Data":"8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb"}
Sep 17 16:28:14.911591 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.911504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t" event={"ID":"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e","Type":"ContainerDied","Data":"7a8e95318c2033bda268b8d901ba973cbd8c80f23d0eb3dac9e01b2fce40d6fc"}
Sep 17 16:28:14.911591 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.911531    2580 scope.go:117] "RemoveContainer" containerID="26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859"
Sep 17 16:28:14.919639 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.919617    2580 scope.go:117] "RemoveContainer" containerID="8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb"
Sep 17 16:28:14.926677 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.926662    2580 scope.go:117] "RemoveContainer" containerID="48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf"
Sep 17 16:28:14.933755 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.933740    2580 scope.go:117] "RemoveContainer" containerID="26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859"
Sep 17 16:28:14.934073 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:28:14.934051    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859\": container with ID starting with 26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859 not found: ID does not exist" containerID="26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859"
Sep 17 16:28:14.934126 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.934082    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859"} err="failed to get container status \"26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859\": rpc error: code = NotFound desc = could not find container \"26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859\": container with ID starting with 26364fae71bbb25550faf9f5d67749872f74ec8935e78ff792dad3c05d6d1859 not found: ID does not exist"
Sep 17 16:28:14.934126 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.934100    2580 scope.go:117] "RemoveContainer" containerID="8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb"
Sep 17 16:28:14.934350 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:28:14.934333    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb\": container with ID starting with 8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb not found: ID does not exist" containerID="8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb"
Sep 17 16:28:14.934420 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.934361    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb"} err="failed to get container status \"8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb\": rpc error: code = NotFound desc = could not find container \"8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb\": container with ID starting with 8e36383839a49a67cfcf31a589c09ca2ae2dcce0ef0a48a81c9d545a0ea5becb not found: ID does not exist"
Sep 17 16:28:14.934420 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.934384    2580 scope.go:117] "RemoveContainer" containerID="48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf"
Sep 17 16:28:14.934627 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:28:14.934611    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf\": container with ID starting with 48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf not found: ID does not exist" containerID="48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf"
Sep 17 16:28:14.934663 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.934631    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf"} err="failed to get container status \"48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf\": rpc error: code = NotFound desc = could not find container \"48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf\": container with ID starting with 48317f441838aa794fbdc355d64455148f8ca5aaaafb8630a83c331233c191bf not found: ID does not exist"
Sep 17 16:28:14.944941 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.944924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tck47\" (UniqueName: \"kubernetes.io/projected/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-api-access-tck47\") pod \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") "
Sep 17 16:28:14.945025 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.944963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-cabundle-cert\") pod \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") "
Sep 17 16:28:14.945025 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.944986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kserve-provision-location\") pod \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") "
Sep 17 16:28:14.945134 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.945091    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-rbac-proxy-sar-config\") pod \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") "
Sep 17 16:28:14.945134 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.945124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-proxy-tls\") pod \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\" (UID: \"1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e\") "
Sep 17 16:28:14.945306 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.945281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" (UID: "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:28:14.945412 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.945363    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:28:14.945412 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.945361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" (UID: "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:14.945412 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.945373    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" (UID: "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:14.946994 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.946974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-api-access-tck47" (OuterVolumeSpecName: "kube-api-access-tck47") pod "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" (UID: "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e"). InnerVolumeSpecName "kube-api-access-tck47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:14.947108 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:14.947086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" (UID: "1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:15.046123 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:15.046075    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:28:15.046123 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:15.046116    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:28:15.046123 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:15.046130    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tck47\" (UniqueName: \"kubernetes.io/projected/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-kube-api-access-tck47\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:28:15.046355 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:15.046143    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:28:15.234943 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:15.234789    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"]
Sep 17 16:28:15.236434 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:15.236408    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75587d66bd-lhn6t"]
Sep 17 16:28:15.674746 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:15.674710    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" path="/var/lib/kubelet/pods/1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e/volumes"
Sep 17 16:28:20.860433 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:20.860401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:22.034455 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.034383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:22.070888 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.070852    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"]
Sep 17 16:28:22.071158 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071146    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kube-rbac-proxy"
Sep 17 16:28:22.071201 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071161    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kube-rbac-proxy"
Sep 17 16:28:22.071201 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071174    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="storage-initializer"
Sep 17 16:28:22.071201 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071179    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="storage-initializer"
Sep 17 16:28:22.071299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071204    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container"
Sep 17 16:28:22.071299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071210    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container"
Sep 17 16:28:22.071299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071217    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kube-rbac-proxy"
Sep 17 16:28:22.071299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071221    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kube-rbac-proxy"
Sep 17 16:28:22.071299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071229    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="storage-initializer"
Sep 17 16:28:22.071299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071234    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="storage-initializer"
Sep 17 16:28:22.071299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071240    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container"
Sep 17 16:28:22.071299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071245    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container"
Sep 17 16:28:22.071538 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071317    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kserve-container"
Sep 17 16:28:22.071538 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071327    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kserve-container"
Sep 17 16:28:22.071538 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071333    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cab384e-c0f5-4465-9fa8-0ce4be3d33e1" containerName="kube-rbac-proxy"
Sep 17 16:28:22.071538 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.071338    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2eefdd-6f9e-4eb1-9574-14ddf2ee232e" containerName="kube-rbac-proxy"
Sep 17 16:28:22.073658 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.073642    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.078189 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.078145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 16:28:22.078347 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.078328    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:28:22.078684 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.078663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:28:22.078817 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.078724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:28:22.078965 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.078948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 16:28:22.079346 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.079329    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 16:28:22.079469 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.079409    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:28:22.092377 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.092351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"]
Sep 17 16:28:22.203919 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.203871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.203919 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.203923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6zpt\" (UniqueName: \"kubernetes.io/projected/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-api-access-b6zpt\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.204191 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.204009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.204191 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.204057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c855d5-d2ac-4eeb-aa72-408ad87887d3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.204191 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.204089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.305512 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.305413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.305512 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.305471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.305512 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.305507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6zpt\" (UniqueName: \"kubernetes.io/projected/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-api-access-b6zpt\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.305792 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.305554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.305792 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.305609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c855d5-d2ac-4eeb-aa72-408ad87887d3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.306003 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.305980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.306150 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.306131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.306244 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.306229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.308200 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.308182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c855d5-d2ac-4eeb-aa72-408ad87887d3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.315824 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.315774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6zpt\" (UniqueName: \"kubernetes.io/projected/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-api-access-b6zpt\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.319380 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.319358    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:22.383391 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.383359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:22.661740 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.661724    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"]
Sep 17 16:28:22.663972 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:28:22.663940    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c855d5_d2ac_4eeb_aa72_408ad87887d3.slice/crio-d3f545dcb527da9b6a80806a94d410d7726ecd48a43df71c9bac08564f8b83d3 WatchSource:0}: Error finding container d3f545dcb527da9b6a80806a94d410d7726ecd48a43df71c9bac08564f8b83d3: Status 404 returned error can't find the container with id d3f545dcb527da9b6a80806a94d410d7726ecd48a43df71c9bac08564f8b83d3
Sep 17 16:28:22.946921 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.946820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" event={"ID":"85c855d5-d2ac-4eeb-aa72-408ad87887d3","Type":"ContainerStarted","Data":"f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d"}
Sep 17 16:28:22.946921 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:22.946885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" event={"ID":"85c855d5-d2ac-4eeb-aa72-408ad87887d3","Type":"ContainerStarted","Data":"d3f545dcb527da9b6a80806a94d410d7726ecd48a43df71c9bac08564f8b83d3"}
Sep 17 16:28:23.952097 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:23.952060    2580 generic.go:356] "Generic (PLEG): container finished" podID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerID="f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d" exitCode=0
Sep 17 16:28:23.952510 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:23.952153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" event={"ID":"85c855d5-d2ac-4eeb-aa72-408ad87887d3","Type":"ContainerDied","Data":"f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d"}
Sep 17 16:28:24.958561 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:24.958524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" event={"ID":"85c855d5-d2ac-4eeb-aa72-408ad87887d3","Type":"ContainerStarted","Data":"5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3"}
Sep 17 16:28:24.959001 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:24.958581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" event={"ID":"85c855d5-d2ac-4eeb-aa72-408ad87887d3","Type":"ContainerStarted","Data":"d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d"}
Sep 17 16:28:24.959001 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:24.958605    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:24.959001 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:24.958625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:24.960089 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:24.960064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 16:28:24.986381 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:24.986324    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podStartSLOduration=1.876741947 podStartE2EDuration="2.986304156s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.109562209s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:24.985248072 +0000 UTC m=+2989.827610867" watchObservedRunningTime="2026-09-17 16:28:24.986304156 +0000 UTC m=+2989.828666999"
Sep 17 16:28:25.962565 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:25.962524    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 16:28:30.967078 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:30.967046    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:28:30.967758 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:30.967719    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 16:28:31.276695 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:31.276594    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:35.826416 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:35.826386    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:28:35.831572 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:35.831543    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:28:36.436618 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:36.436586    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:36.856100 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:36.856018    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:28:40.967972 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:40.967934    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 16:28:50.967901 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:28:50.967854    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 16:29:00.967680 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:00.967639    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 16:29:10.967702 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:10.967658    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 16:29:20.968068 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:20.968024    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 16:29:30.967824 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:30.967713    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 16:29:40.967971 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:40.967934    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:29:42.006511 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:42.006476    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"]
Sep 17 16:29:42.006999 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:42.006773    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container" containerID="cri-o://d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d" gracePeriod=30
Sep 17 16:29:42.006999 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:42.006854    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kube-rbac-proxy" containerID="cri-o://5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3" gracePeriod=30
Sep 17 16:29:42.022082 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:42.022057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:29:42.274441 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:42.274348    2580 generic.go:356] "Generic (PLEG): container finished" podID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerID="5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3" exitCode=2
Sep 17 16:29:42.274441 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:42.274421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" event={"ID":"85c855d5-d2ac-4eeb-aa72-408ad87887d3","Type":"ContainerDied","Data":"5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3"}
Sep 17 16:29:43.070594 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.070561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:29:43.094743 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.094708    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"]
Sep 17 16:29:43.098089 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.098071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.100152 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.100127    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 16:29:43.100294 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.100130    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 16:29:43.110813 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.110770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"]
Sep 17 16:29:43.248880 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.248850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:29:43.268073 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.268031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.268266 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.268088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.268266 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.268164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvrk\" (UniqueName: \"kubernetes.io/projected/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-api-access-bwvrk\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.268266 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.268193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.368663 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.368624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.368663 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.368676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.368998 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.368708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvrk\" (UniqueName: \"kubernetes.io/projected/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-api-access-bwvrk\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.368998 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.368726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.369170 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.369146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.369380 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.369358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.371422 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.371398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.377707 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.377685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvrk\" (UniqueName: \"kubernetes.io/projected/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-api-access-bwvrk\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.408488 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.408450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:43.683028 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:43.683003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"]
Sep 17 16:29:43.685701 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:29:43.685666    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6bc97f_f7ce_4c2e_b389_8dabc924a00b.slice/crio-d81874022fb44510333f04d2bbf54acf82cfa3c8da67b9d5571ee818c5cc0972 WatchSource:0}: Error finding container d81874022fb44510333f04d2bbf54acf82cfa3c8da67b9d5571ee818c5cc0972: Status 404 returned error can't find the container with id d81874022fb44510333f04d2bbf54acf82cfa3c8da67b9d5571ee818c5cc0972
Sep 17 16:29:44.283279 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:44.283233    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph" event={"ID":"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b","Type":"ContainerStarted","Data":"dca73be370652eb793140c8794593bf94d40f8f712cc2572572baf179edbfa23"}
Sep 17 16:29:44.283793 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:44.283304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph" event={"ID":"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b","Type":"ContainerStarted","Data":"d81874022fb44510333f04d2bbf54acf82cfa3c8da67b9d5571ee818c5cc0972"}
Sep 17 16:29:45.963253 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:45.963210    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 16:29:47.152596 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.152568    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:29:47.301105 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.300976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-rbac-proxy-sar-config\") pod \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") "
Sep 17 16:29:47.301105 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-cabundle-cert\") pod \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") "
Sep 17 16:29:47.301105 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301062    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kserve-provision-location\") pod \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") "
Sep 17 16:29:47.301105 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301095    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6zpt\" (UniqueName: \"kubernetes.io/projected/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-api-access-b6zpt\") pod \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") "
Sep 17 16:29:47.301445 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c855d5-d2ac-4eeb-aa72-408ad87887d3-proxy-tls\") pod \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\" (UID: \"85c855d5-d2ac-4eeb-aa72-408ad87887d3\") "
Sep 17 16:29:47.301445 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85c855d5-d2ac-4eeb-aa72-408ad87887d3" (UID: "85c855d5-d2ac-4eeb-aa72-408ad87887d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:29:47.301550 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301449    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85c855d5-d2ac-4eeb-aa72-408ad87887d3" (UID: "85c855d5-d2ac-4eeb-aa72-408ad87887d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:29:47.301550 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301450    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85c855d5-d2ac-4eeb-aa72-408ad87887d3" (UID: "85c855d5-d2ac-4eeb-aa72-408ad87887d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:29:47.301663 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301574    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:29:47.301663 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301587    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c855d5-d2ac-4eeb-aa72-408ad87887d3-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:29:47.301663 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.301597    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:29:47.303553 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.303523    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-api-access-b6zpt" (OuterVolumeSpecName: "kube-api-access-b6zpt") pod "85c855d5-d2ac-4eeb-aa72-408ad87887d3" (UID: "85c855d5-d2ac-4eeb-aa72-408ad87887d3"). InnerVolumeSpecName "kube-api-access-b6zpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:29:47.303685 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.303607    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c855d5-d2ac-4eeb-aa72-408ad87887d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85c855d5-d2ac-4eeb-aa72-408ad87887d3" (UID: "85c855d5-d2ac-4eeb-aa72-408ad87887d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:29:47.304525 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.304502    2580 generic.go:356] "Generic (PLEG): container finished" podID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerID="d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d" exitCode=0
Sep 17 16:29:47.304610 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.304555    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"
Sep 17 16:29:47.304656 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.304552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" event={"ID":"85c855d5-d2ac-4eeb-aa72-408ad87887d3","Type":"ContainerDied","Data":"d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d"}
Sep 17 16:29:47.304698 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.304679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng" event={"ID":"85c855d5-d2ac-4eeb-aa72-408ad87887d3","Type":"ContainerDied","Data":"d3f545dcb527da9b6a80806a94d410d7726ecd48a43df71c9bac08564f8b83d3"}
Sep 17 16:29:47.304737 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.304704    2580 scope.go:117] "RemoveContainer" containerID="5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3"
Sep 17 16:29:47.318272 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.318249    2580 scope.go:117] "RemoveContainer" containerID="d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d"
Sep 17 16:29:47.326049 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.326027    2580 scope.go:117] "RemoveContainer" containerID="f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d"
Sep 17 16:29:47.331062 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.331037    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"]
Sep 17 16:29:47.334079 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.334058    2580 scope.go:117] "RemoveContainer" containerID="5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3"
Sep 17 16:29:47.334384 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:29:47.334364    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3\": container with ID starting with 5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3 not found: ID does not exist" containerID="5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3"
Sep 17 16:29:47.334445 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.334395    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3"} err="failed to get container status \"5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3\": rpc error: code = NotFound desc = could not find container \"5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3\": container with ID starting with 5e13a1e037cd85d8538b347566d1b75c409268094d3358ba9899a70a1d52b6f3 not found: ID does not exist"
Sep 17 16:29:47.334445 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.334414    2580 scope.go:117] "RemoveContainer" containerID="d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d"
Sep 17 16:29:47.334696 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:29:47.334679    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d\": container with ID starting with d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d not found: ID does not exist" containerID="d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d"
Sep 17 16:29:47.334747 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.334699    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d"} err="failed to get container status \"d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d\": rpc error: code = NotFound desc = could not find container \"d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d\": container with ID starting with d43f3058e335c27de9cc78eff48e4c0a6397fda000cd9aa8fdf5a72cb3281b7d not found: ID does not exist"
Sep 17 16:29:47.334747 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.334713    2580 scope.go:117] "RemoveContainer" containerID="f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d"
Sep 17 16:29:47.335066 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:29:47.335049    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d\": container with ID starting with f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d not found: ID does not exist" containerID="f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d"
Sep 17 16:29:47.335130 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.335070    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d"} err="failed to get container status \"f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d\": rpc error: code = NotFound desc = could not find container \"f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d\": container with ID starting with f543f78cd5d41d7745407a9e9d06762208180df1f4ba066fe9f3d9e7b504705d not found: ID does not exist"
Sep 17 16:29:47.339817 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.339782    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7fd98fc9f5-zkhng"]
Sep 17 16:29:47.402904 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.402872    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6zpt\" (UniqueName: \"kubernetes.io/projected/85c855d5-d2ac-4eeb-aa72-408ad87887d3-kube-api-access-b6zpt\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:29:47.402904 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.402898    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c855d5-d2ac-4eeb-aa72-408ad87887d3-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:29:47.674973 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:47.674939    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" path="/var/lib/kubelet/pods/85c855d5-d2ac-4eeb-aa72-408ad87887d3/volumes"
Sep 17 16:29:50.319603 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:50.319528    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph_8c6bc97f-f7ce-4c2e-b389-8dabc924a00b/storage-initializer/0.log"
Sep 17 16:29:50.319603 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:50.319566    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" containerID="dca73be370652eb793140c8794593bf94d40f8f712cc2572572baf179edbfa23" exitCode=1
Sep 17 16:29:50.320112 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:50.319609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph" event={"ID":"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b","Type":"ContainerDied","Data":"dca73be370652eb793140c8794593bf94d40f8f712cc2572572baf179edbfa23"}
Sep 17 16:29:51.325498 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:51.325465    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph_8c6bc97f-f7ce-4c2e-b389-8dabc924a00b/storage-initializer/0.log"
Sep 17 16:29:51.326051 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:51.325526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph" event={"ID":"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b","Type":"ContainerStarted","Data":"a71451d1c8151b6debc87a1a5eba22ccbcb8a9262d9cc3f429d43c09cc0f069f"}
Sep 17 16:29:53.108192 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.108156    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"]
Sep 17 16:29:53.108684 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.108398    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph" podUID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" containerName="storage-initializer" containerID="cri-o://a71451d1c8151b6debc87a1a5eba22ccbcb8a9262d9cc3f429d43c09cc0f069f" gracePeriod=30
Sep 17 16:29:53.132541 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.132480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:29:53.334390 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.334366    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph_8c6bc97f-f7ce-4c2e-b389-8dabc924a00b/storage-initializer/1.log"
Sep 17 16:29:53.334885 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.334867    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph_8c6bc97f-f7ce-4c2e-b389-8dabc924a00b/storage-initializer/0.log"
Sep 17 16:29:53.334964 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.334906    2580 generic.go:356] "Generic (PLEG): container finished" podID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" containerID="a71451d1c8151b6debc87a1a5eba22ccbcb8a9262d9cc3f429d43c09cc0f069f" exitCode=1
Sep 17 16:29:53.335016 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.334987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph" event={"ID":"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b","Type":"ContainerDied","Data":"a71451d1c8151b6debc87a1a5eba22ccbcb8a9262d9cc3f429d43c09cc0f069f"}
Sep 17 16:29:53.335088 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.335075    2580 scope.go:117] "RemoveContainer" containerID="dca73be370652eb793140c8794593bf94d40f8f712cc2572572baf179edbfa23"
Sep 17 16:29:53.348703 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.348685    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph_8c6bc97f-f7ce-4c2e-b389-8dabc924a00b/storage-initializer/1.log"
Sep 17 16:29:53.352095 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.352079    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:53.448722 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.448617    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-rbac-proxy-sar-config\") pod \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") "
Sep 17 16:29:53.448722 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.448658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-proxy-tls\") pod \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") "
Sep 17 16:29:53.448722 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.448696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kserve-provision-location\") pod \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") "
Sep 17 16:29:53.448722 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.448719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwvrk\" (UniqueName: \"kubernetes.io/projected/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-api-access-bwvrk\") pod \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\" (UID: \"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b\") "
Sep 17 16:29:53.449116 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.448982    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" (UID: "8c6bc97f-f7ce-4c2e-b389-8dabc924a00b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:29:53.449116 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.449086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" (UID: "8c6bc97f-f7ce-4c2e-b389-8dabc924a00b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:29:53.451070 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.451038    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-api-access-bwvrk" (OuterVolumeSpecName: "kube-api-access-bwvrk") pod "8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" (UID: "8c6bc97f-f7ce-4c2e-b389-8dabc924a00b"). InnerVolumeSpecName "kube-api-access-bwvrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:29:53.451182 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.451089    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" (UID: "8c6bc97f-f7ce-4c2e-b389-8dabc924a00b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:29:53.549178 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.549127    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:29:53.549178 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.549172    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:29:53.549178 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.549182    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:29:53.549178 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:53.549192    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwvrk\" (UniqueName: \"kubernetes.io/projected/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b-kube-api-access-bwvrk\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:29:54.215468 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.215434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:29:54.234223 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234177    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"]
Sep 17 16:29:54.234550 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234531    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" containerName="storage-initializer"
Sep 17 16:29:54.234644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234553    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" containerName="storage-initializer"
Sep 17 16:29:54.234644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234567    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container"
Sep 17 16:29:54.234644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234575    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container"
Sep 17 16:29:54.234644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234586    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" containerName="storage-initializer"
Sep 17 16:29:54.234644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234595    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" containerName="storage-initializer"
Sep 17 16:29:54.234644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234615    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="storage-initializer"
Sep 17 16:29:54.234644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234624    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="storage-initializer"
Sep 17 16:29:54.234644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234636    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kube-rbac-proxy"
Sep 17 16:29:54.234644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234645    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kube-rbac-proxy"
Sep 17 16:29:54.235113 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234719    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kube-rbac-proxy"
Sep 17 16:29:54.235113 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234730    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c855d5-d2ac-4eeb-aa72-408ad87887d3" containerName="kserve-container"
Sep 17 16:29:54.235113 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234739    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" containerName="storage-initializer"
Sep 17 16:29:54.235113 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.234748    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" containerName="storage-initializer"
Sep 17 16:29:54.239361 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.239335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.241452 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.241424    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:29:54.241610 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.241497    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:29:54.241705 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.241683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 16:29:54.241844 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.241824    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 16:29:54.249215 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.249189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"]
Sep 17 16:29:54.254990 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.254961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8718bcd-5ef2-461c-acad-add633fb544a-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.255130 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.254998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.255130 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.255023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.255130 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.255067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbfj\" (UniqueName: \"kubernetes.io/projected/c8718bcd-5ef2-461c-acad-add633fb544a-kube-api-access-6nbfj\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.255130 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.255118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8718bcd-5ef2-461c-acad-add633fb544a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.340190 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.340164    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph_8c6bc97f-f7ce-4c2e-b389-8dabc924a00b/storage-initializer/1.log"
Sep 17 16:29:54.340361 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.340259    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"
Sep 17 16:29:54.340361 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.340260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph" event={"ID":"8c6bc97f-f7ce-4c2e-b389-8dabc924a00b","Type":"ContainerDied","Data":"d81874022fb44510333f04d2bbf54acf82cfa3c8da67b9d5571ee818c5cc0972"}
Sep 17 16:29:54.340438 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.340391    2580 scope.go:117] "RemoveContainer" containerID="a71451d1c8151b6debc87a1a5eba22ccbcb8a9262d9cc3f429d43c09cc0f069f"
Sep 17 16:29:54.356157 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.356136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8718bcd-5ef2-461c-acad-add633fb544a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.356263 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.356182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8718bcd-5ef2-461c-acad-add633fb544a-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.356263 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.356204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.356383 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:29:54.356291    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 16:29:54.356383 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.356302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.356383 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:29:54.356358    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8718bcd-5ef2-461c-acad-add633fb544a-proxy-tls podName:c8718bcd-5ef2-461c-acad-add633fb544a nodeName:}" failed. No retries permitted until 2026-09-17 16:29:54.856336472 +0000 UTC m=+3079.698699254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c8718bcd-5ef2-461c-acad-add633fb544a-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" (UID: "c8718bcd-5ef2-461c-acad-add633fb544a") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 16:29:54.356540 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.356386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbfj\" (UniqueName: \"kubernetes.io/projected/c8718bcd-5ef2-461c-acad-add633fb544a-kube-api-access-6nbfj\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.356540 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.356526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8718bcd-5ef2-461c-acad-add633fb544a-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.356902 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.356883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.356941 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.356916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.368975 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.368942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbfj\" (UniqueName: \"kubernetes.io/projected/c8718bcd-5ef2-461c-acad-add633fb544a-kube-api-access-6nbfj\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.372211 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.372183    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"]
Sep 17 16:29:54.377238 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.377213    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-8685b689b9-kdgph"]
Sep 17 16:29:54.411931 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.411899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:29:54.860119 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.860074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8718bcd-5ef2-461c-acad-add633fb544a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:54.862760 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:54.862726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8718bcd-5ef2-461c-acad-add633fb544a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:55.150667 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:55.150567    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:29:55.438299 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:55.437977    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"]
Sep 17 16:29:55.440783 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:29:55.440757    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8718bcd_5ef2_461c_acad_add633fb544a.slice/crio-de04a3c8c185e1c91aff36260b3c4c76be971f39299821c7899188978ad57bf1 WatchSource:0}: Error finding container de04a3c8c185e1c91aff36260b3c4c76be971f39299821c7899188978ad57bf1: Status 404 returned error can't find the container with id de04a3c8c185e1c91aff36260b3c4c76be971f39299821c7899188978ad57bf1
Sep 17 16:29:55.676937 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:55.676896    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c6bc97f-f7ce-4c2e-b389-8dabc924a00b" path="/var/lib/kubelet/pods/8c6bc97f-f7ce-4c2e-b389-8dabc924a00b/volumes"
Sep 17 16:29:56.351192 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:56.351157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" event={"ID":"c8718bcd-5ef2-461c-acad-add633fb544a","Type":"ContainerStarted","Data":"bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282"}
Sep 17 16:29:56.351375 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:29:56.351220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" event={"ID":"c8718bcd-5ef2-461c-acad-add633fb544a","Type":"ContainerStarted","Data":"de04a3c8c185e1c91aff36260b3c4c76be971f39299821c7899188978ad57bf1"}
Sep 17 16:30:00.368232 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:00.368192    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8718bcd-5ef2-461c-acad-add633fb544a" containerID="bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282" exitCode=0
Sep 17 16:30:00.368704 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:00.368246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" event={"ID":"c8718bcd-5ef2-461c-acad-add633fb544a","Type":"ContainerDied","Data":"bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282"}
Sep 17 16:30:08.411474 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:08.411435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" event={"ID":"c8718bcd-5ef2-461c-acad-add633fb544a","Type":"ContainerStarted","Data":"95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8"}
Sep 17 16:30:08.411970 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:08.411530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" event={"ID":"c8718bcd-5ef2-461c-acad-add633fb544a","Type":"ContainerStarted","Data":"d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f"}
Sep 17 16:30:08.411970 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:08.411773    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:30:08.432262 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:08.432215    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podStartSLOduration=3.24721789 podStartE2EDuration="14.432199937s" totalImagesPullingTime="7.312408288s" totalInitContainerRuntime="3.872573759s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:30:08.431113404 +0000 UTC m=+3093.273476211" watchObservedRunningTime="2026-09-17 16:30:08.432199937 +0000 UTC m=+3093.274562742"
Sep 17 16:30:09.415955 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:09.415917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:30:09.416953 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:09.416925    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:30:10.420437 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:10.420394    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:30:15.424649 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:15.424621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:30:15.430259 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:15.430221    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:30:25.425638 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:25.425537    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:30:28.503135 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:28.503103    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:33.606925 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:33.606889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:34.091609 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:34.091569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:30:35.425220 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:35.425181    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:30:45.426024 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:45.425982    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:30:55.425974 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:30:55.425933    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:31:05.425450 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:05.425407    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:31:15.425292 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:15.425251    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:31:25.425469 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:25.425424    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:31:26.670520 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:26.670473    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:31:36.670943 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:36.670912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:31:45.995169 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:45.995132    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"]
Sep 17 16:31:45.995607 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:45.995383    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" containerID="cri-o://d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f" gracePeriod=30
Sep 17 16:31:45.995607 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:45.995416    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kube-rbac-proxy" containerID="cri-o://95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8" gracePeriod=30
Sep 17 16:31:46.025026 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:46.025001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:31:46.671023 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:46.670974    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 16:31:46.793642 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:46.793603    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8718bcd-5ef2-461c-acad-add633fb544a" containerID="95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8" exitCode=2
Sep 17 16:31:46.793836 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:46.793672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" event={"ID":"c8718bcd-5ef2-461c-acad-add633fb544a","Type":"ContainerDied","Data":"95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8"}
Sep 17 16:31:50.438792 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.438764    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:31:50.518669 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.518636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nbfj\" (UniqueName: \"kubernetes.io/projected/c8718bcd-5ef2-461c-acad-add633fb544a-kube-api-access-6nbfj\") pod \"c8718bcd-5ef2-461c-acad-add633fb544a\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") "
Sep 17 16:31:50.518889 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.518675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-kube-rbac-proxy-sar-config\") pod \"c8718bcd-5ef2-461c-acad-add633fb544a\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") "
Sep 17 16:31:50.518889 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.518706    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-cabundle-cert\") pod \"c8718bcd-5ef2-461c-acad-add633fb544a\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") "
Sep 17 16:31:50.518889 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.518729    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8718bcd-5ef2-461c-acad-add633fb544a-proxy-tls\") pod \"c8718bcd-5ef2-461c-acad-add633fb544a\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") "
Sep 17 16:31:50.518889 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.518783    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8718bcd-5ef2-461c-acad-add633fb544a-kserve-provision-location\") pod \"c8718bcd-5ef2-461c-acad-add633fb544a\" (UID: \"c8718bcd-5ef2-461c-acad-add633fb544a\") "
Sep 17 16:31:50.519160 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.519104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8718bcd-5ef2-461c-acad-add633fb544a" (UID: "c8718bcd-5ef2-461c-acad-add633fb544a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:31:50.519219 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.519153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8718bcd-5ef2-461c-acad-add633fb544a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8718bcd-5ef2-461c-acad-add633fb544a" (UID: "c8718bcd-5ef2-461c-acad-add633fb544a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:50.519219 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.519164    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8718bcd-5ef2-461c-acad-add633fb544a" (UID: "c8718bcd-5ef2-461c-acad-add633fb544a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:31:50.521093 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.521067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8718bcd-5ef2-461c-acad-add633fb544a-kube-api-access-6nbfj" (OuterVolumeSpecName: "kube-api-access-6nbfj") pod "c8718bcd-5ef2-461c-acad-add633fb544a" (UID: "c8718bcd-5ef2-461c-acad-add633fb544a"). InnerVolumeSpecName "kube-api-access-6nbfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:31:50.521093 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.521080    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8718bcd-5ef2-461c-acad-add633fb544a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8718bcd-5ef2-461c-acad-add633fb544a" (UID: "c8718bcd-5ef2-461c-acad-add633fb544a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:31:50.619908 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.619875    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbfj\" (UniqueName: \"kubernetes.io/projected/c8718bcd-5ef2-461c-acad-add633fb544a-kube-api-access-6nbfj\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:31:50.619908 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.619902    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:31:50.619908 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.619913    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8718bcd-5ef2-461c-acad-add633fb544a-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:31:50.620144 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.619922    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8718bcd-5ef2-461c-acad-add633fb544a-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:31:50.620144 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.619930    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8718bcd-5ef2-461c-acad-add633fb544a-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:31:50.810545 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.810454    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8718bcd-5ef2-461c-acad-add633fb544a" containerID="d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f" exitCode=0
Sep 17 16:31:50.810545 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.810508    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"
Sep 17 16:31:50.810721 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.810509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" event={"ID":"c8718bcd-5ef2-461c-acad-add633fb544a","Type":"ContainerDied","Data":"d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f"}
Sep 17 16:31:50.810721 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.810657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" event={"ID":"c8718bcd-5ef2-461c-acad-add633fb544a","Type":"ContainerDied","Data":"de04a3c8c185e1c91aff36260b3c4c76be971f39299821c7899188978ad57bf1"}
Sep 17 16:31:50.810721 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.810698    2580 scope.go:117] "RemoveContainer" containerID="95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8"
Sep 17 16:31:50.819525 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.819506    2580 scope.go:117] "RemoveContainer" containerID="d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f"
Sep 17 16:31:50.826750 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.826733    2580 scope.go:117] "RemoveContainer" containerID="bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282"
Sep 17 16:31:50.832569 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.832542    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"]
Sep 17 16:31:50.834241 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.834228    2580 scope.go:117] "RemoveContainer" containerID="95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8"
Sep 17 16:31:50.834509 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:31:50.834491    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8\": container with ID starting with 95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8 not found: ID does not exist" containerID="95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8"
Sep 17 16:31:50.834552 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.834518    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8"} err="failed to get container status \"95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8\": rpc error: code = NotFound desc = could not find container \"95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8\": container with ID starting with 95e7ba083c3c3280560771281d8a8f275600a3312913bd80911b9f43c097e8f8 not found: ID does not exist"
Sep 17 16:31:50.834552 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.834535    2580 scope.go:117] "RemoveContainer" containerID="d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f"
Sep 17 16:31:50.834814 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:31:50.834783    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f\": container with ID starting with d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f not found: ID does not exist" containerID="d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f"
Sep 17 16:31:50.834867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.834827    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f"} err="failed to get container status \"d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f\": rpc error: code = NotFound desc = could not find container \"d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f\": container with ID starting with d393b060c028d822c963138c2c1e0a8a544a027f1f78df63c6885d5970a6e57f not found: ID does not exist"
Sep 17 16:31:50.834867 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.834841    2580 scope.go:117] "RemoveContainer" containerID="bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282"
Sep 17 16:31:50.835058 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:31:50.835044    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282\": container with ID starting with bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282 not found: ID does not exist" containerID="bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282"
Sep 17 16:31:50.835095 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.835062    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282"} err="failed to get container status \"bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282\": rpc error: code = NotFound desc = could not find container \"bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282\": container with ID starting with bcf24f86f7d93754a5bfcc0eb5b0f2ebe84c3f69bcdf80e1ea4f426b65eea282 not found: ID does not exist"
Sep 17 16:31:50.839333 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:50.839313    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp"]
Sep 17 16:31:51.422115 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:51.422063    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-j68bp" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": context deadline exceeded"
Sep 17 16:31:51.674836 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:31:51.674744    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" path="/var/lib/kubelet/pods/c8718bcd-5ef2-461c-acad-add633fb544a/volumes"
Sep 17 16:32:15.499294 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:15.499256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:20.596026 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.595995    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:20.655405 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.655369    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"]
Sep 17 16:32:20.655697 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.655684    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container"
Sep 17 16:32:20.655746 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.655700    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container"
Sep 17 16:32:20.655746 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.655719    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="storage-initializer"
Sep 17 16:32:20.655746 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.655729    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="storage-initializer"
Sep 17 16:32:20.655746 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.655741    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kube-rbac-proxy"
Sep 17 16:32:20.655746 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.655746    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kube-rbac-proxy"
Sep 17 16:32:20.655938 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.655823    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kserve-container"
Sep 17 16:32:20.655938 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.655830    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8718bcd-5ef2-461c-acad-add633fb544a" containerName="kube-rbac-proxy"
Sep 17 16:32:20.658049 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.658033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.660180 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.660144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:32:20.660566 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.660546    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:32:20.660675 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.660628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-859eae-predictor-serving-cert\""
Sep 17 16:32:20.660675 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.660628    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-859eae-kube-rbac-proxy-sar-config\""
Sep 17 16:32:20.660843 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.660714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:32:20.660959 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.660940    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:32:20.661081 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.661012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:32:20.670488 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.670465    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"]
Sep 17 16:32:20.720565 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.720529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:32:20.761399 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.761360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-cabundle-cert\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.761559 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.761511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-proxy-tls\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.761620 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.761576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kserve-provision-location\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.761731 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.761716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcndl\" (UniqueName: \"kubernetes.io/projected/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-api-access-jcndl\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.761824 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.761763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-rbac-proxy-sar-config\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.862209 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.862172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-cabundle-cert\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.862406 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.862219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-proxy-tls\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.862406 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.862291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kserve-provision-location\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.862406 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.862335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcndl\" (UniqueName: \"kubernetes.io/projected/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-api-access-jcndl\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.862406 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.862358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-rbac-proxy-sar-config\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.862847 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.862789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kserve-provision-location\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.863112 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.863058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-rbac-proxy-sar-config\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.863112 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.863060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-cabundle-cert\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.864903 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.864883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-proxy-tls\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.871647 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.871625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcndl\" (UniqueName: \"kubernetes.io/projected/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-api-access-jcndl\") pod \"isvc-primary-859eae-predictor-684899d844-qpnjm\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") " pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:20.968943 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:20.968896    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:21.241376 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:21.241321    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"]
Sep 17 16:32:21.244258 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:32:21.244230    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfed06a_73a9_4c1b_bbd0_6ba20d130dc0.slice/crio-dc685d743bb8a4f3ff0ea6b93b574b2c9bd4ad427b406536e04e5287b51c71cb WatchSource:0}: Error finding container dc685d743bb8a4f3ff0ea6b93b574b2c9bd4ad427b406536e04e5287b51c71cb: Status 404 returned error can't find the container with id dc685d743bb8a4f3ff0ea6b93b574b2c9bd4ad427b406536e04e5287b51c71cb
Sep 17 16:32:21.931474 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:21.931427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" event={"ID":"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0","Type":"ContainerStarted","Data":"afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5"}
Sep 17 16:32:21.931946 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:21.931501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" event={"ID":"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0","Type":"ContainerStarted","Data":"dc685d743bb8a4f3ff0ea6b93b574b2c9bd4ad427b406536e04e5287b51c71cb"}
Sep 17 16:32:25.948663 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:25.948627    2580 generic.go:356] "Generic (PLEG): container finished" podID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerID="afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5" exitCode=0
Sep 17 16:32:25.949094 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:25.948676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" event={"ID":"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0","Type":"ContainerDied","Data":"afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5"}
Sep 17 16:32:26.955086 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:26.954998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" event={"ID":"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0","Type":"ContainerStarted","Data":"3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf"}
Sep 17 16:32:26.955086 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:26.955060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" event={"ID":"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0","Type":"ContainerStarted","Data":"9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f"}
Sep 17 16:32:26.955587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:26.955325    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:26.955587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:26.955473    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:26.957024 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:26.956993    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 16:32:26.974651 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:26.974593    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podStartSLOduration=3.177454402 podStartE2EDuration="6.974575278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.797120876s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:32:26.973190919 +0000 UTC m=+3231.815553739" watchObservedRunningTime="2026-09-17 16:32:26.974575278 +0000 UTC m=+3231.816938082"
Sep 17 16:32:27.959148 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:27.959106    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 16:32:32.964414 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:32.964376    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:32:32.965049 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:32.965021    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 16:32:42.965760 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:42.965714    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 16:32:52.965658 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:32:52.965607    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 16:33:02.964979 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:33:02.964935    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 16:33:12.964971 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:33:12.964928    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 16:33:22.965563 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:33:22.965518    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 16:33:32.965432 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:33:32.965389    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 16:33:35.731457 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:33:35.731423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:33:35.851300 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:33:35.851272    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:33:35.857023 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:33:35.856999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:33:42.966169 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:33:42.966138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:34:05.875816 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:05.875772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:06.317072 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:06.316988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:23.967465 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:23.967434    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"]
Sep 17 16:34:23.967914 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:23.967687    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container" containerID="cri-o://9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f" gracePeriod=30
Sep 17 16:34:23.967914 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:23.967731    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kube-rbac-proxy" containerID="cri-o://3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf" gracePeriod=30
Sep 17 16:34:24.416986 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:24.416946    2580 generic.go:356] "Generic (PLEG): container finished" podID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerID="3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf" exitCode=2
Sep 17 16:34:24.417161 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:24.417004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" event={"ID":"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0","Type":"ContainerDied","Data":"3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf"}
Sep 17 16:34:24.817506 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:24.817425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:24.880659 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:24.880626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:27.959728 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:27.959683    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 17 16:34:27.973515 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:27.973485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:29.121018 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.120991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:34:29.172640 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.172611    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kserve-provision-location\") pod \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") "
Sep 17 16:34:29.172835 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.172657    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-proxy-tls\") pod \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") "
Sep 17 16:34:29.172835 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.172690    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-rbac-proxy-sar-config\") pod \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") "
Sep 17 16:34:29.172835 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.172710    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-cabundle-cert\") pod \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") "
Sep 17 16:34:29.172835 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.172735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcndl\" (UniqueName: \"kubernetes.io/projected/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-api-access-jcndl\") pod \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\" (UID: \"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0\") "
Sep 17 16:34:29.173092 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.173018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" (UID: "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:29.173199 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.173169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" (UID: "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:29.173199 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.173182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" (UID: "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:29.175038 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.175016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" (UID: "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:34:29.175038 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.175022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-api-access-jcndl" (OuterVolumeSpecName: "kube-api-access-jcndl") pod "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" (UID: "cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0"). InnerVolumeSpecName "kube-api-access-jcndl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:34:29.273461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.273363    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:34:29.273461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.273401    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:34:29.273461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.273411    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcndl\" (UniqueName: \"kubernetes.io/projected/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kube-api-access-jcndl\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:34:29.273461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.273420    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:34:29.273461 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.273429    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0-proxy-tls\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 17 16:34:29.438399 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.438364    2580 generic.go:356] "Generic (PLEG): container finished" podID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerID="9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f" exitCode=0
Sep 17 16:34:29.438566 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.438427    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"
Sep 17 16:34:29.438566 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.438450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" event={"ID":"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0","Type":"ContainerDied","Data":"9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f"}
Sep 17 16:34:29.438566 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.438512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm" event={"ID":"cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0","Type":"ContainerDied","Data":"dc685d743bb8a4f3ff0ea6b93b574b2c9bd4ad427b406536e04e5287b51c71cb"}
Sep 17 16:34:29.438566 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.438540    2580 scope.go:117] "RemoveContainer" containerID="3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf"
Sep 17 16:34:29.447876 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.447859    2580 scope.go:117] "RemoveContainer" containerID="9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f"
Sep 17 16:34:29.456779 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.456756    2580 scope.go:117] "RemoveContainer" containerID="afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5"
Sep 17 16:34:29.461795 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.461770    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"]
Sep 17 16:34:29.464606 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.464586    2580 scope.go:117] "RemoveContainer" containerID="3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf"
Sep 17 16:34:29.465076 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:34:29.465051    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf\": container with ID starting with 3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf not found: ID does not exist" containerID="3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf"
Sep 17 16:34:29.465193 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.465087    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf"} err="failed to get container status \"3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf\": rpc error: code = NotFound desc = could not find container \"3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf\": container with ID starting with 3658eede8401dcd379a8fabbedb8a46d2dcbd59e17a4271fabd785d12a7117cf not found: ID does not exist"
Sep 17 16:34:29.465193 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.465148    2580 scope.go:117] "RemoveContainer" containerID="9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f"
Sep 17 16:34:29.465872 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:34:29.465780    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f\": container with ID starting with 9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f not found: ID does not exist" containerID="9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f"
Sep 17 16:34:29.465998 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.465876    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f"} err="failed to get container status \"9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f\": rpc error: code = NotFound desc = could not find container \"9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f\": container with ID starting with 9da050071e2e48ba36319672448599e58a22d56af3f944b575afba60dbc0732f not found: ID does not exist"
Sep 17 16:34:29.465998 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.465897    2580 scope.go:117] "RemoveContainer" containerID="afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5"
Sep 17 16:34:29.466231 ip-10-0-128-144 kubenswrapper[2580]: E0917 16:34:29.466211    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5\": container with ID starting with afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5 not found: ID does not exist" containerID="afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5"
Sep 17 16:34:29.466293 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.466234    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5"} err="failed to get container status \"afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5\": rpc error: code = NotFound desc = could not find container \"afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5\": container with ID starting with afdbf482325c337ca3a2da9f889e478d6f86a766522f5518cd7b760af91fe5e5 not found: ID does not exist"
Sep 17 16:34:29.467141 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.467121    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-859eae-predictor-684899d844-qpnjm"]
Sep 17 16:34:29.675429 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:29.675394    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" path="/var/lib/kubelet/pods/cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0/volumes"
Sep 17 16:34:30.363054 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:30.363024    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:34:30.424981 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:34:30.424948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:07.784581 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.784548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:07.920897 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.920861    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bp66r/must-gather-j8qcc"]
Sep 17 16:35:07.921223 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.921207    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="storage-initializer"
Sep 17 16:35:07.921312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.921227    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="storage-initializer"
Sep 17 16:35:07.921312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.921252    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container"
Sep 17 16:35:07.921312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.921263    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container"
Sep 17 16:35:07.921312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.921279    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kube-rbac-proxy"
Sep 17 16:35:07.921312 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.921289    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kube-rbac-proxy"
Sep 17 16:35:07.921573 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.921361    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kserve-container"
Sep 17 16:35:07.921573 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.921372    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbfed06a-73a9-4c1b-bbd0-6ba20d130dc0" containerName="kube-rbac-proxy"
Sep 17 16:35:07.924522 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.924498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp66r/must-gather-j8qcc"
Sep 17 16:35:07.926711 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.926677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bp66r\"/\"default-dockercfg-sz5vk\""
Sep 17 16:35:07.926870 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.926836    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bp66r\"/\"openshift-service-ca.crt\""
Sep 17 16:35:07.926930 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.926879    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bp66r\"/\"kube-root-ca.crt\""
Sep 17 16:35:07.934620 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.934595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bp66r/must-gather-j8qcc"]
Sep 17 16:35:07.940480 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:07.940456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:08.040488 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.040401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:08.092764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.092720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b19ca27-25c2-43fa-b068-aea466b037b4-must-gather-output\") pod \"must-gather-j8qcc\" (UID: \"2b19ca27-25c2-43fa-b068-aea466b037b4\") " pod="openshift-must-gather-bp66r/must-gather-j8qcc"
Sep 17 16:35:08.092764 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.092761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpr4\" (UniqueName: \"kubernetes.io/projected/2b19ca27-25c2-43fa-b068-aea466b037b4-kube-api-access-cmpr4\") pod \"must-gather-j8qcc\" (UID: \"2b19ca27-25c2-43fa-b068-aea466b037b4\") " pod="openshift-must-gather-bp66r/must-gather-j8qcc"
Sep 17 16:35:08.194106 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.194067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b19ca27-25c2-43fa-b068-aea466b037b4-must-gather-output\") pod \"must-gather-j8qcc\" (UID: \"2b19ca27-25c2-43fa-b068-aea466b037b4\") " pod="openshift-must-gather-bp66r/must-gather-j8qcc"
Sep 17 16:35:08.194106 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.194106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpr4\" (UniqueName: \"kubernetes.io/projected/2b19ca27-25c2-43fa-b068-aea466b037b4-kube-api-access-cmpr4\") pod \"must-gather-j8qcc\" (UID: \"2b19ca27-25c2-43fa-b068-aea466b037b4\") " pod="openshift-must-gather-bp66r/must-gather-j8qcc"
Sep 17 16:35:08.194417 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.194398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b19ca27-25c2-43fa-b068-aea466b037b4-must-gather-output\") pod \"must-gather-j8qcc\" (UID: \"2b19ca27-25c2-43fa-b068-aea466b037b4\") " pod="openshift-must-gather-bp66r/must-gather-j8qcc"
Sep 17 16:35:08.203086 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.203058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpr4\" (UniqueName: \"kubernetes.io/projected/2b19ca27-25c2-43fa-b068-aea466b037b4-kube-api-access-cmpr4\") pod \"must-gather-j8qcc\" (UID: \"2b19ca27-25c2-43fa-b068-aea466b037b4\") " pod="openshift-must-gather-bp66r/must-gather-j8qcc"
Sep 17 16:35:08.234029 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.233993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp66r/must-gather-j8qcc"
Sep 17 16:35:08.502477 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.502405    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bp66r/must-gather-j8qcc"]
Sep 17 16:35:08.505181 ip-10-0-128-144 kubenswrapper[2580]: W0917 16:35:08.505151    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b19ca27_25c2_43fa_b068_aea466b037b4.slice/crio-8520b60a139323df99eeb664e3581fa350b156e8c7d3e7086620b9866812a340 WatchSource:0}: Error finding container 8520b60a139323df99eeb664e3581fa350b156e8c7d3e7086620b9866812a340: Status 404 returned error can't find the container with id 8520b60a139323df99eeb664e3581fa350b156e8c7d3e7086620b9866812a340
Sep 17 16:35:08.507175 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.507159    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:35:08.597501 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:08.597468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp66r/must-gather-j8qcc" event={"ID":"2b19ca27-25c2-43fa-b068-aea466b037b4","Type":"ContainerStarted","Data":"8520b60a139323df99eeb664e3581fa350b156e8c7d3e7086620b9866812a340"}
Sep 17 16:35:10.607088 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:10.607049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp66r/must-gather-j8qcc" event={"ID":"2b19ca27-25c2-43fa-b068-aea466b037b4","Type":"ContainerStarted","Data":"c5ac0d6f888f732f7415595934935fc97f6a7d0e2d3e9f871d10cb755ff20ab9"}
Sep 17 16:35:10.607605 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:10.607102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp66r/must-gather-j8qcc" event={"ID":"2b19ca27-25c2-43fa-b068-aea466b037b4","Type":"ContainerStarted","Data":"e4e775552843fd9d768e27ea7dc2c0a4355d04be375e0f580caba44e2c844032"}
Sep 17 16:35:10.627682 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:10.627628    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bp66r/must-gather-j8qcc" podStartSLOduration=2.169309447 podStartE2EDuration="3.627610807s" totalImagesPullingTime="1.45830136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:35:10.624658268 +0000 UTC m=+3395.467021073" watchObservedRunningTime="2026-09-17 16:35:10.627610807 +0000 UTC m=+3395.469973650"
Sep 17 16:35:11.814917 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:11.814887    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j755r_b4624302-fc21-4d0e-8c9a-fdfd494389b3/global-pull-secret-syncer/0.log"
Sep 17 16:35:11.881735 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:11.881704    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log"
Sep 17 16:35:12.012529 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:12.012489    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-144.ec2.internal_37310a8518ae4b4c8f438099a305077e/haproxy/0.log"
Sep 17 16:35:13.824587 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:13.824496    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:14.347125 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:14.347065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mrl_a7c82c64-126e-4dce-9f44-a1279a7de1e4/konnectivity-agent/0.log" limit=1024
Sep 17 16:35:16.632069 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:16.632034    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vsbdp_7015c765-eb52-4355-a95c-54bc96a4ac9a/node-exporter/0.log"
Sep 17 16:35:16.662613 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:16.662549    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vsbdp_7015c765-eb52-4355-a95c-54bc96a4ac9a/kube-rbac-proxy/0.log"
Sep 17 16:35:16.689246 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:16.689218    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vsbdp_7015c765-eb52-4355-a95c-54bc96a4ac9a/init-textfile/0.log"
Sep 17 16:35:19.366484 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.366447    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/2.log"
Sep 17 16:35:19.369789 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.369766    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k7mhd_2037b600-0a62-4629-93a6-16b83614e117/console-operator/3.log"
Sep 17 16:35:19.944958 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.944924    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"]
Sep 17 16:35:19.951033 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.950995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:19.973281 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.973245    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"]
Sep 17 16:35:19.982446 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.982412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-sys\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:19.982446 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.982443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f8sf\" (UniqueName: \"kubernetes.io/projected/a63d8073-06a6-4e96-b387-77eb2b48a961-kube-api-access-5f8sf\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:19.982670 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.982477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-lib-modules\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:19.982670 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.982538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-proc\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:19.982670 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:19.982611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-podres\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.083369 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.083328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-sys\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.083571 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.083379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f8sf\" (UniqueName: \"kubernetes.io/projected/a63d8073-06a6-4e96-b387-77eb2b48a961-kube-api-access-5f8sf\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.083571 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.083425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-lib-modules\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.083571 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.083454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-sys\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.083738 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.083575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-proc\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.083738 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.083593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-lib-modules\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.083738 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.083655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-proc\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.083738 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.083697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-podres\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.083938 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.083849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a63d8073-06a6-4e96-b387-77eb2b48a961-podres\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.099777 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.099748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f8sf\" (UniqueName: \"kubernetes.io/projected/a63d8073-06a6-4e96-b387-77eb2b48a961-kube-api-access-5f8sf\") pod \"perf-node-gather-daemonset-knjj5\" (UID: \"a63d8073-06a6-4e96-b387-77eb2b48a961\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.264923 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.264841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:20.529453 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.529384    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-7xgj4_2b7ac916-0662-4e55-860d-da12c759654f/volume-data-source-validator/0.log"
Sep 17 16:35:20.546163 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.546124    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"]
Sep 17 16:35:20.665887 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:20.665857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5" event={"ID":"a63d8073-06a6-4e96-b387-77eb2b48a961","Type":"ContainerStarted","Data":"b04f7c269a57e2da4916c761f8cb846d580e3b6dadb38b4ea8f2653e35ea357d"}
Sep 17 16:35:21.480848 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:21.480817    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdtrg_38dc1beb-42f8-4732-bc67-a5a9f39a97e6/dns/0.log"
Sep 17 16:35:21.506779 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:21.506746    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdtrg_38dc1beb-42f8-4732-bc67-a5a9f39a97e6/kube-rbac-proxy/0.log"
Sep 17 16:35:21.673948 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:21.673919    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hv6gp_6ecaf429-99cd-48a9-8eaf-96cdc04dd17b/dns-node-resolver/0.log"
Sep 17 16:35:21.675350 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:21.675329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:21.675503 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:21.675384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5" event={"ID":"a63d8073-06a6-4e96-b387-77eb2b48a961","Type":"ContainerStarted","Data":"1e9b4c94facf10fd7a69eb901760409a06675829c635ca2bd7dd895ce3c41fd0"}
Sep 17 16:35:21.688954 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:21.688901    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5" podStartSLOduration=2.688884529 podStartE2EDuration="2.688884529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:35:21.6877756 +0000 UTC m=+3406.530138427" watchObservedRunningTime="2026-09-17 16:35:21.688884529 +0000 UTC m=+3406.531247333"
Sep 17 16:35:22.237431 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:22.237402    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rwm2w_62b44d6a-61ff-458c-917c-0f4a105a381d/node-ca/0.log"
Sep 17 16:35:23.548397 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:23.548365    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zzkfw_08c92ab5-a08c-4d43-9cb6-d20f69be2a2c/serve-healthcheck-canary/0.log"
Sep 17 16:35:24.017260 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:24.017221    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9jg52_0c959272-5175-4063-bf12-e0b9b8e38650/insights-operator/0.log"
Sep 17 16:35:24.017932 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:24.017915    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9jg52_0c959272-5175-4063-bf12-e0b9b8e38650/insights-operator/1.log"
Sep 17 16:35:24.176036 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:24.176003    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vfpmt_bfec7824-0193-4d02-a151-e149f7bb3d61/kube-rbac-proxy/0.log"
Sep 17 16:35:24.212704 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:24.212679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vfpmt_bfec7824-0193-4d02-a151-e149f7bb3d61/exporter/0.log"
Sep 17 16:35:24.252106 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:24.252074    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vfpmt_bfec7824-0193-4d02-a151-e149f7bb3d61/extractor/0.log"
Sep 17 16:35:27.226006 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:27.225922    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-nmgt6_7d52f465-876d-4092-b2b4-31e5119b5fd7/manager/0.log"
Sep 17 16:35:27.456586 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:27.456550    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-cj7ql_2842e953-41b7-4eb3-8e8b-877129e52bf6/seaweedfs/0.log"
Sep 17 16:35:27.489597 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:27.489522    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-sw9j5_c0f2c7a9-3e90-4088-81ab-d15bcbbf76b1/seaweedfs-tls-custom/0.log"
Sep 17 16:35:27.520507 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:27.520480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-wr9rh_36caaf15-691b-42cb-b30b-5962cb72e9ec/seaweedfs-tls-serving/0.log"
Sep 17 16:35:27.688492 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:27.688454    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-knjj5"
Sep 17 16:35:32.559041 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:32.559009    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-rl5dj_a6762993-a05b-4825-8b15-c77c144e43e3/kube-storage-version-migrator-operator/1.log"
Sep 17 16:35:32.560644 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:32.560621    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-rl5dj_a6762993-a05b-4825-8b15-c77c144e43e3/kube-storage-version-migrator-operator/0.log"
Sep 17 16:35:33.613063 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:33.613030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5fm5g_8e9cd823-b4d5-4fb8-8a9f-4c4f28a170c2/kube-multus/0.log"
Sep 17 16:35:33.688063 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:33.688036    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2t7_0814fe57-0b03-485c-bd00-544675e5ec2c/kube-multus-additional-cni-plugins/0.log"
Sep 17 16:35:33.716070 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:33.716008    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2t7_0814fe57-0b03-485c-bd00-544675e5ec2c/egress-router-binary-copy/0.log"
Sep 17 16:35:33.743170 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:33.743144    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2t7_0814fe57-0b03-485c-bd00-544675e5ec2c/cni-plugins/0.log"
Sep 17 16:35:33.772079 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:33.772053    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2t7_0814fe57-0b03-485c-bd00-544675e5ec2c/bond-cni-plugin/0.log"
Sep 17 16:35:33.814568 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:33.814535    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2t7_0814fe57-0b03-485c-bd00-544675e5ec2c/routeoverride-cni/0.log"
Sep 17 16:35:33.842178 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:33.842142    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2t7_0814fe57-0b03-485c-bd00-544675e5ec2c/whereabouts-cni-bincopy/0.log"
Sep 17 16:35:33.874840 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:33.874755    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8t2t7_0814fe57-0b03-485c-bd00-544675e5ec2c/whereabouts-cni/0.log"
Sep 17 16:35:34.560274 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:34.560225    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xztjr_d86c0b23-dea0-497e-a3ca-fa1037439372/network-metrics-daemon/0.log"
Sep 17 16:35:34.591216 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:34.591188    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xztjr_d86c0b23-dea0-497e-a3ca-fa1037439372/kube-rbac-proxy/0.log"
Sep 17 16:35:36.058773 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:36.058746    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpzbl_ce14c586-76d1-4bf1-8f04-f261b9afc9ce/ovn-controller/0.log"
Sep 17 16:35:36.093596 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:36.093569    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpzbl_ce14c586-76d1-4bf1-8f04-f261b9afc9ce/ovn-acl-logging/0.log"
Sep 17 16:35:36.136061 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:36.136034    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpzbl_ce14c586-76d1-4bf1-8f04-f261b9afc9ce/kube-rbac-proxy-node/0.log"
Sep 17 16:35:36.176459 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:36.176428    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpzbl_ce14c586-76d1-4bf1-8f04-f261b9afc9ce/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 16:35:36.206028 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:36.206004    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpzbl_ce14c586-76d1-4bf1-8f04-f261b9afc9ce/northd/0.log"
Sep 17 16:35:36.238407 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:36.238372    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpzbl_ce14c586-76d1-4bf1-8f04-f261b9afc9ce/nbdb/0.log"
Sep 17 16:35:36.268780 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:36.268756    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpzbl_ce14c586-76d1-4bf1-8f04-f261b9afc9ce/sbdb/0.log"
Sep 17 16:35:36.324762 ip-10-0-128-144 kubenswrapper[2580]: I0917 16:35:36.324685    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpzbl_ce14c586-76d1-4bf1-8f04-f261b9afc9ce/ovnkube-controller/0.log"