Sep 16 18:44:29.204562 ip-10-0-139-120 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:44:29.647051 ip-10-0-139-120 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:29.647051 ip-10-0-139-120 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:44:29.647051 ip-10-0-139-120 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:29.647051 ip-10-0-139-120 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:44:29.647051 ip-10-0-139-120 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:29.649036 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.648949    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:44:29.654392 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654369    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.654392 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654386    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.654392 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654391    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.654392 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654395    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.654392 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654398    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654401    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654404    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654407    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654410    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654413    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654416    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654419    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654422    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654424    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654427    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654429    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654432    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654435    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654437    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654440    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654445    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654448    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654451    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654454    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.654604 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654456    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654459    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654461    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654464    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654466    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654468    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654471    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654473    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654476    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654479    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654481    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654484    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654486    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654489    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654492    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654495    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654497    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654500    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654503    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654505    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.655092 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654509    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654514    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654516    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654519    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654522    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654525    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654527    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654530    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654532    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654535    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654538    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654540    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654543    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654547    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654550    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654554    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654558    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654561    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654564    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.655600 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654567    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654570    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654573    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654576    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654578    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654581    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654584    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654587    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654590    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654592    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654595    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654598    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654600    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654603    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654606    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654609    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654611    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654614    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654617    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654620    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.656059 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654623    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654626    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654629    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654631    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654634    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654642    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654646    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654649    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654652    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654655    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654658    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654661    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654663    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654666    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654669    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.654671    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655363    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655370    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655375    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.656566 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655379    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655383    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655386    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655388    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655391    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655394    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655396    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655399    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655402    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655404    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655408    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655412    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655416    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655419    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655423    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655425    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655429    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655432    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655434    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.657027 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655437    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655440    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655442    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655445    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655449    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655451    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655454    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655456    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655459    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655461    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655464    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655466    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655469    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655474    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655477    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655480    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655482    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655485    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655487    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655490    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.657528 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655492    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655495    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655497    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655500    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655503    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655505    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655508    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655510    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655513    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655516    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655518    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655521    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655523    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655526    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655528    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655531    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655534    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655537    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655539    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655542    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.658025 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655544    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655547    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655549    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655552    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655555    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655558    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655561    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655564    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655567    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655569    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655573    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655577    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655580    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655583    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655586    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655589    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655592    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655594    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655597    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655600    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.658539 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655603    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655607    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655610    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655613    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655616    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655618    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655621    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655623    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655626    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655628    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655631    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655634    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655636    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655639    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655641    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655644    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.655646    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656018    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656023    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656026    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.659040 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656029    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656032    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656035    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656038    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656042    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656046    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656049    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656052    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656055    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656057    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656060    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656063    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656065    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656071    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656074    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656077    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656079    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656082    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656084    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.659532 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656087    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656089    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656092    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656095    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656097    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656100    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656103    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656105    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656108    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656110    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656113    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656117    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656120    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656122    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656125    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656127    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656130    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656132    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656135    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.660015 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656137    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656140    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656142    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656145    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656148    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656151    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656153    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656157    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656160    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656163    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656165    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656168    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656170    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656173    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656175    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656178    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656180    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656182    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656185    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656188    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.660582 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656190    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656192    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656195    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656198    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656201    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656203    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656206    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656208    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656211    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656213    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656216    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656218    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656221    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656223    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656226    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656228    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656231    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656234    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656238    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.661069 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656242    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656247    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656250    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656252    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656255    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656258    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656260    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656276    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656279    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656281    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656284    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656287    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656289    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656292    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656295    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656297    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656300    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656302    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.656305    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656377    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656410    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:44:29.661592 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656416    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656421    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656425    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656428    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656432    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656437    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656440    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656443    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656447    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656451    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656454    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656459    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656462    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656465    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656469    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656472    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656477    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656480    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656483    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656486    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656489    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656493    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656496    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:44:29.662119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656499    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656503    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656510    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656513    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656516    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656519    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656522    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656526    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656529    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656532    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656534    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656537    2584 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656540    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656544    2584 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656547    2584 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656550    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656553    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656556    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656559    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656562    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656566    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656569    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656573    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656576    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656580    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:44:29.662741 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656583    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656586    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656589    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656592    2584 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656596    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656600    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656603    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656606    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656622    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656626    2584 flags.go:64] FLAG: --help="false"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656630    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656634    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656637    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656641    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656644    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656648    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656651    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656653    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656656    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656659    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656662    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656665    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656668    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656671    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:44:29.663384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656674    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656677    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656680    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656683    2584 flags.go:64] FLAG: --lock-file=""
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656686    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656689    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656692    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656697    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656701    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656703    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656706    2584 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656709    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656712    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656718    2584 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656721    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656725    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656728    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656732    2584 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656735    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656738    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656741    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656744    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656747    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656750    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656753    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:44:29.663986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656760    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656763    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656767    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656770    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656773    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656778    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656780    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656784    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656786    2584 flags.go:64] FLAG: --port="10250"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656789    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656793    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ddd962cbf62d9083"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656796    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656799    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656802    2584 flags.go:64] FLAG: --register-node="true"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656805    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656809    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656813    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656816    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656818    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656822    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656825    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656829    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656833    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656835    2584 flags.go:64] FLAG: --runonce="false"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656838    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:44:29.664616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656841    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656844    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656847    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656850    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656853    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656856    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656859    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656862    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656865    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656868    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656871    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656874    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656876    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656879    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656884    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656887    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656890    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656895    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656898    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656901    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656904    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656907    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656910    2584 flags.go:64] FLAG: --v="2"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656914    2584 flags.go:64] FLAG: --version="false"
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656919    2584 flags.go:64] FLAG: --vmodule=""
Sep 16 18:44:29.665220 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656923    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.656926    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657038    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657043    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657047    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657049    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657052    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657055    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657057    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657060    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657062    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657065    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657068    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657071    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657073    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657076    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657079    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657083    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657086    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.665859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657090    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657093    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657095    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657098    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657101    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657104    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657107    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657110    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657112    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657115    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657117    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657120    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657123    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657129    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657132    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657134    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657138    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657141    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657143    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657146    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.666382 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657149    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657153    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657156    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657159    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657161    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657164    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657167    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657169    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657172    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657174    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657177    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657180    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657183    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657185    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657188    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657191    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657194    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657197    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657200    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.666875 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657203    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657207    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657210    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657214    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657217    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657219    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657222    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657226    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657228    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657233    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657235    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657238    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657240    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657243    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657246    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657248    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657251    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657253    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657277    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.667376 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657280    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657282    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657285    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657288    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657291    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657293    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657296    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657299    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657301    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657304    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657307    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657311    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657313    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657315    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657318    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657321    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657324    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657326    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657329    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657332    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.667859 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657334    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657337    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657341    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.657344    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.658070    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.666129    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.666146    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.667031    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.667039    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667104    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667110    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667113    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667116    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667119    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667122    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.668375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667125    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667127    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667131    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667133    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667136    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667139    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667141    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667144    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667147    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667150    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667153    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667155    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667158    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667161    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667163    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667166    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667169    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667171    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667174    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667176    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.668772 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667179    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667182    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667185    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667188    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667191    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667194    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667196    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667199    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667202    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667204    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667207    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667210    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667212    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667215    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667218    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667220    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667223    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667226    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667228    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667231    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.669259 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667233    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667236    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667239    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667242    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667244    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667247    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667249    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667252    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667254    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667257    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667261    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667278    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667281    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667285    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667287    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667290    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667293    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667296    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667299    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667302    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.669811 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667304    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667307    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667311    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667314    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667317    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667320    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667323    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667326    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667329    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667333    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667336    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667339    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667341    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667345    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667347    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667350    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667353    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667356    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.670341 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667358    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667361    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667363    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667370    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667372    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667375    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667378    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667381    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667384    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667387    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667390    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667393    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667395    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667398    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667401    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.670824 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.667406    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667536    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667540    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667543    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667546    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667549    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667552    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667554    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667557    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667559    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667562    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667565    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667568    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667571    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667573    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667576    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667578    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667581    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667584    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667586    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:29.671198 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667589    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667591    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667594    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667597    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667599    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667603    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667607    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667610    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667613    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667616    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667619    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667622    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667624    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667627    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667629    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667632    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667634    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667637    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667640    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667642    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:29.671724 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667645    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667647    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667650    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667652    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667655    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667659    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667663    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667666    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667669    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667671    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667674    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667676    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667679    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667681    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667684    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667687    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667689    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667692    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667694    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:29.672226 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667697    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667700    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667702    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667711    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667714    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667716    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667719    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667722    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667724    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667727    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667729    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667732    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667734    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667737    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667739    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667742    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667746    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667749    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667752    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:29.672804 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667755    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667758    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667761    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667763    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667766    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667768    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667771    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667773    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667776    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667778    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667781    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667783    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667786    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667789    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667791    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667794    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667796    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667805    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667807    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667810    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:29.673284 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667812    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:29.673769 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:29.667815    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:29.673769 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.667820    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:29.673769 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.667941    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:44:29.673769 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.670380    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:44:29.673769 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.671535    2584 server.go:1019] "Starting client certificate rotation"
Sep 16 18:44:29.673769 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.671643    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:29.673769 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.672451    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:29.674520 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.674505    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:29.680967 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.680943    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:29.691059 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.691041    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:44:29.694460 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.694445    2584 log.go:25] "Validated CRI v1 image API"
Sep 16 18:44:29.695234 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.695218    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:44:29.698721 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.698704    2584 fs.go:135] Filesystem UUIDs: map[21296afc-ed14-47cc-9399-6fbc3fe4f63f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8b51b2a0-f5ee-4b3e-98e1-d1b38858fd04:/dev/nvme0n1p4]
Sep 16 18:44:29.698788 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.698721    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:44:29.704086 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.703973    2584 manager.go:216] Machine: {Timestamp:2026-09-16 18:44:29.702482226 +0000 UTC m=+0.369421378 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092081 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f902b8743d0bfc7b9dbba72b3197d SystemUUID:ec2f902b-8743-d0bf-c7b9-dbba72b3197d BootID:75fe5f2f-ec3f-48bf-95f5-3dee4bc64242 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:d4:4c:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:d4:4c:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:d5:0f:a6:c4:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:44:29.704086 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.704078    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:44:29.704196 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.704161    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:44:29.705964 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.705947    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:29.706101 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.706074    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:44:29.706238 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.706107    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:44:29.707682 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.707672    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:44:29.707721 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.707687    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:44:29.707721 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.707696    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:29.708538 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.708527    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:29.709870 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.709860    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:44:29.714487 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.714475    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:29.714604 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.714594    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:44:29.714648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.714633    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:44:29.714677 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.714658    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:44:29.714677 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.714676    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:44:29.714749 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.714685    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:44:29.714749 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.714698    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:44:29.715135 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.715122    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:29.715177 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.715148    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:29.717511 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.717496    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:44:29.720514 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.720500    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:44:29.720568 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.720530    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:44:29.723465 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723449    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:44:29.723533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723470    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:44:29.723533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723478    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:44:29.723533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723483    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:44:29.723533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723492    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:44:29.723533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723497    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:44:29.723533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723510    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:44:29.723533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723515    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:44:29.723533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723523    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:44:29.723533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723528    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:44:29.723761 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723542    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:44:29.723761 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.723548    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:44:29.724285 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.724260    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:44:29.724332 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.724285    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:44:29.725056 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.725035    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:44:29.725056 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.725046    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:44:29.728156 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.728144    2584 server.go:1268] "Started kubelet"
Sep 16 18:44:29.728226 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.728200    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:44:29.728365 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.728285    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:44:29.728443 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.728387    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:44:29.729037 ip-10-0-139-120 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:44:29.732214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.732187    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:44:29.732464 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.732442    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w2ndg"
Sep 16 18:44:29.733965 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.733892    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:44:29.737502 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.737464    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:44:29.737562 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.737539    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-120.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:44:29.737826 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.737812    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w2ndg"
Sep 16 18:44:29.738719 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.737584    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-120.ec2.internal.18d5e16964c57636  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-120.ec2.internal,UID:ip-10-0-139-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-120.ec2.internal,},FirstTimestamp:2026-09-16 18:44:29.728118326 +0000 UTC m=+0.395057462,LastTimestamp:2026-09-16 18:44:29.728118326 +0000 UTC m=+0.395057462,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-120.ec2.internal,}"
Sep 16 18:44:29.745058 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.745041    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:29.745766 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.745748    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:44:29.745918 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.745900    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:29.746522 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746505    2584 factory.go:55] Registering systemd factory
Sep 16 18:44:29.746593 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746524    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:44:29.746645 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746605    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:44:29.746645 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746608    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:44:29.746645 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746628    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:44:29.746783 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746691    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:44:29.746783 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746701    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:44:29.746783 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746740    2584 factory.go:153] Registering CRI-O factory
Sep 16 18:44:29.746783 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746751    2584 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:44:29.746946 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746852    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:44:29.746946 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746870    2584 factory.go:103] Registering Raw factory
Sep 16 18:44:29.746946 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.746886    2584 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:44:29.747285 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.747257    2584 manager.go:349] Starting recovery of all containers
Sep 16 18:44:29.748605 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.748584    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:29.751612 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.751589    2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-120.ec2.internal\" not found" node="ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.757375 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.757361    2584 manager.go:354] Recovery completed
Sep 16 18:44:29.761824 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.761812    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.763322 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.763309    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.763382 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.763338    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.763382 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.763348    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.763824 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.763812    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:44:29.763824 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.763823    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.763909 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.763824    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:44:29.763909 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.763857    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:29.764478 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.764462    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.764520 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.764491    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.764897 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.764887    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.766059 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.766044    2584 policy_none.go:49] "None policy: Start"
Sep 16 18:44:29.766059 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.766061    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:44:29.766165 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.766071    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:44:29.768016 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.768004    2584 policy_none.go:47] "Start"
Sep 16 18:44:29.793537 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.793520    2584 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.793572    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.793585    2584 server.go:87] "Starting device plugin registration server"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.793837    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.793849    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.794006    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.794069    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.794076    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.794589    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.794660    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.808708    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.809867    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.809918    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.809942    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.809986    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:44:29.820214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.812260    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:29.894994 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.894943    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.896056 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.896035    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.896165 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.896073    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.896165 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.896091    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.896165 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.896121    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.902719 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.902672    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.902719 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.902694    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-120.ec2.internal\": node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:29.910805 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.910787    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal"]
Sep 16 18:44:29.910889 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.910847    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.912368 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.912352    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.912420 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.912382    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.912420 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.912396    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.914721 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.914708    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.914850 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.914825    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.914942 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.914863    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.915472 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.915456    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.915562 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.915487    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.915562 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.915501    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.916148 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.916127    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.916237 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.916161    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.916237 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.916175    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.918393 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.918377    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.918484 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.918405    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:29.919093 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.919079    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:29.919156 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.919104    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:29.919156 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.919115    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:29.920468 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.920455    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:29.948722 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.948703    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62aae6dbe28b0afbce02d1018e37007d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal\" (UID: \"62aae6dbe28b0afbce02d1018e37007d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.948826 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.948734    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62aae6dbe28b0afbce02d1018e37007d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal\" (UID: \"62aae6dbe28b0afbce02d1018e37007d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.948826 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:29.948761    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8611bbd0ccde75189058c67a0a76ec36-config\") pod \"kube-apiserver-proxy-ip-10-0-139-120.ec2.internal\" (UID: \"8611bbd0ccde75189058c67a0a76ec36\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.951906 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.951892    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-120.ec2.internal\" not found" node="ip-10-0-139-120.ec2.internal"
Sep 16 18:44:29.956301 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:29.956285    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-120.ec2.internal\" not found" node="ip-10-0-139-120.ec2.internal"
Sep 16 18:44:30.021120 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.021094    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.049870 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.049846    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62aae6dbe28b0afbce02d1018e37007d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal\" (UID: \"62aae6dbe28b0afbce02d1018e37007d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:30.049948 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.049873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62aae6dbe28b0afbce02d1018e37007d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal\" (UID: \"62aae6dbe28b0afbce02d1018e37007d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:30.049948 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.049891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8611bbd0ccde75189058c67a0a76ec36-config\") pod \"kube-apiserver-proxy-ip-10-0-139-120.ec2.internal\" (UID: \"8611bbd0ccde75189058c67a0a76ec36\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:30.050014 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.049942    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62aae6dbe28b0afbce02d1018e37007d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal\" (UID: \"62aae6dbe28b0afbce02d1018e37007d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:30.050014 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.049959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62aae6dbe28b0afbce02d1018e37007d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal\" (UID: \"62aae6dbe28b0afbce02d1018e37007d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:30.050014 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.049942    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8611bbd0ccde75189058c67a0a76ec36-config\") pod \"kube-apiserver-proxy-ip-10-0-139-120.ec2.internal\" (UID: \"8611bbd0ccde75189058c67a0a76ec36\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:30.121674 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.121644    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.221912 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.221824    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.255310 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.255285    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:30.260709 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.260424    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:30.322865 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.322698    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.423376 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.423331    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.523607 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.523532    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.627389 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.627366    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.671227 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.671187    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:44:30.671665 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.671368    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:30.671665 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.671379    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:30.727834 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.727810    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.744626 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.744597    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:39:30 +0000 UTC" deadline="2028-02-22 06:49:28.270940987 +0000 UTC"
Sep 16 18:44:30.744626 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.744624    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12564h4m57.526320225s"
Sep 16 18:44:30.828417 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.828396    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.930571 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:30.930528    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:30.990445 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:30.990415    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62aae6dbe28b0afbce02d1018e37007d.slice/crio-8b9e14acd2e86a812ce40cbb6b9526a59ab6fcac6f92186a49c6be724c6e0054 WatchSource:0}: Error finding container 8b9e14acd2e86a812ce40cbb6b9526a59ab6fcac6f92186a49c6be724c6e0054: Status 404 returned error can't find the container with id 8b9e14acd2e86a812ce40cbb6b9526a59ab6fcac6f92186a49c6be724c6e0054
Sep 16 18:44:30.990673 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:30.990648    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8611bbd0ccde75189058c67a0a76ec36.slice/crio-1debf864179de2cc678d3bafc1db82dceb253550035c7f77ef7633e9e112847b WatchSource:0}: Error finding container 1debf864179de2cc678d3bafc1db82dceb253550035c7f77ef7633e9e112847b: Status 404 returned error can't find the container with id 1debf864179de2cc678d3bafc1db82dceb253550035c7f77ef7633e9e112847b
Sep 16 18:44:30.994916 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:30.994893    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:44:31.030905 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.030868    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:31.136347 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.136251    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-120.ec2.internal\" not found"
Sep 16 18:44:31.168612 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.168587    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:31.246167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.246139    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:31.259464 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.259444    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:31.261037 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.261023    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal"
Sep 16 18:44:31.273711 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.273688    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:31.283354 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.283337    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:31.718331 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.718294    2584 apiserver.go:52] "Watching apiserver"
Sep 16 18:44:31.726556 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.726524    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:44:31.727050 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.727020    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal","openshift-multus/multus-additional-cni-plugins-hj6rj","openshift-ovn-kubernetes/ovnkube-node-lpxk7","kube-system/global-pull-secret-syncer-6cn4c","kube-system/konnectivity-agent-w9rjm","kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal","openshift-image-registry/node-ca-dlqjl","openshift-multus/multus-wh7hw","openshift-multus/network-metrics-daemon-lcxfx","openshift-network-diagnostics/network-check-target-cvrx8","openshift-network-operator/iptables-alerter-k8glm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc","openshift-cluster-node-tuning-operator/tuned-g2rsd","openshift-dns/node-resolver-hh97b"]
Sep 16 18:44:31.729737 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.729709    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.732367 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.732029    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.732750 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.732728    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:44:31.732844 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.732728    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.732844 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.732777    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.733089 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.733071    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rqqgb\""
Sep 16 18:44:31.735018 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.734647    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.735018 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.734896    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqwrj\""
Sep 16 18:44:31.735281 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.735263    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.735382 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.735265    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.737044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.736978    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:31.737160 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.737059    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:31.737678 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.737324    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.737678 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.737382    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:44:31.737678 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.737525    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:44:31.737678 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.737543    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:44:31.737678 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.737557    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lk9mx\""
Sep 16 18:44:31.737678 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.737652    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.738104 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.737687    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:44:31.739478 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.739456    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:31.741990 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.741969    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9qnb6\""
Sep 16 18:44:31.742720 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.742211    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.742720 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.742301    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:44:31.742720 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.742671    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:44:31.744368 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.744348    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:31.744464 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.744437    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:31.744643 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.744549    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.745054 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.744912    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.745054 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.745005    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4bfkc\""
Sep 16 18:44:31.745829 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.745799    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:31.747511 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.746821    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.751068 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.750830    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.752113 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.752082    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.752204 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.752177    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:44:31.752270 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.752224    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:44:31.752577 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.752518    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.752726 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.752091    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p8nvl\""
Sep 16 18:44:31.754876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.754589    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:31.754876 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.754688    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:31.755240 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.755222    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:44:31.755581 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.755564    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:44:31.755714 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.755564    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-96s9g\""
Sep 16 18:44:31.757243 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.757220    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.758769 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.758744    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:31.759643 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.759622    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.760907    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-run-netns\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.760944    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-var-lib-kubelet\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.760974    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-run-multus-certs\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761001    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d14846-8078-4eed-972f-75e7fff7d7cb-konnectivity-ca\") pod \"konnectivity-agent-w9rjm\" (UID: \"d5d14846-8078-4eed-972f-75e7fff7d7cb\") " pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761026    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-sysctl-d\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761049    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-run\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761074    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-lib-modules\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761100    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-slash\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761127    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6764d66c-f9f6-4990-b83c-85216eaae28d-ovnkube-script-lib\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761154    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-run-netns\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761179    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-sysconfig\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-run-ovn\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761228    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-cnibin\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761253    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.761400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761252    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-etc-kubernetes\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.762279 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761578    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nxv64\""
Sep 16 18:44:31.762279 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761730    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:44:31.762279 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.761980    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.762477 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762341    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91214ab9-3932-439b-821e-5746a112e114-dbus\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:31.762477 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762376    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-sys\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.762477 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6764d66c-f9f6-4990-b83c-85216eaae28d-ovnkube-config\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.762477 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6764d66c-f9f6-4990-b83c-85216eaae28d-ovn-node-metrics-cert\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.762647 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762495    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg26z\" (UniqueName: \"kubernetes.io/projected/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-kube-api-access-xg26z\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:31.762647 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762532    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-cni-binary-copy\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.762647 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762573    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-systemd-units\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.762647 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762620    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-var-lib-openvswitch\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.762835 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762655    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-run-ovn-kubernetes\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.762835 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762683    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.762835 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762711    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:31.762835 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-kubernetes\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.762835 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762766    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-sysctl-conf\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.762835 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762790    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-kubelet\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.762835 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762822    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91214ab9-3932-439b-821e-5746a112e114-kubelet-config\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-socket-dir-parent\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762871    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-var-lib-cni-multus\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762895    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c17acc3f-e899-405c-aa12-55aa19fc7b86-tmp\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762921    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-daemon-config\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc2d6470-4c6a-4a11-9387-5519d9858bc2-host\") pod \"node-ca-dlqjl\" (UID: \"dc2d6470-4c6a-4a11-9387-5519d9858bc2\") " pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762966    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-tuned\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.762992    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-run-openvswitch\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763016    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-cni-bin\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763113    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-run-k8s-cni-cncf-io\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.763183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763139    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-var-lib-cni-bin\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763244    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-conf-dir\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763271    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z967m\" (UniqueName: \"kubernetes.io/projected/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-kube-api-access-z967m\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2chb\" (UniqueName: \"kubernetes.io/projected/c17acc3f-e899-405c-aa12-55aa19fc7b86-kube-api-access-n2chb\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763413    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-log-socket\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763440    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77743e7d-141f-40d6-9fa8-c58c087a0a6a-hosts-file\") pod \"node-resolver-hh97b\" (UID: \"77743e7d-141f-40d6-9fa8-c58c087a0a6a\") " pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763492    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-host\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763520    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-etc-openvswitch\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763545    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6764d66c-f9f6-4990-b83c-85216eaae28d-env-overrides\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763575    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7n7j\" (UniqueName: \"kubernetes.io/projected/77743e7d-141f-40d6-9fa8-c58c087a0a6a-kube-api-access-k7n7j\") pod \"node-resolver-hh97b\" (UID: \"77743e7d-141f-40d6-9fa8-c58c087a0a6a\") " pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763625    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763652    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-var-lib-kubelet\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763676    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-node-log\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.763704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763699    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-hostroot\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763745    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc2d6470-4c6a-4a11-9387-5519d9858bc2-serviceca\") pod \"node-ca-dlqjl\" (UID: \"dc2d6470-4c6a-4a11-9387-5519d9858bc2\") " pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763791    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vtx\" (UniqueName: \"kubernetes.io/projected/6764d66c-f9f6-4990-b83c-85216eaae28d-kube-api-access-x2vtx\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763816    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77743e7d-141f-40d6-9fa8-c58c087a0a6a-tmp-dir\") pod \"node-resolver-hh97b\" (UID: \"77743e7d-141f-40d6-9fa8-c58c087a0a6a\") " pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763841    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-cni-dir\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763864    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-modprobe-d\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763885    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-systemd\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763919    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-cni-netd\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.763949    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-run-systemd\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.764001    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-system-cni-dir\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.764027    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpfjz\" (UniqueName: \"kubernetes.io/projected/dc2d6470-4c6a-4a11-9387-5519d9858bc2-kube-api-access-jpfjz\") pod \"node-ca-dlqjl\" (UID: \"dc2d6470-4c6a-4a11-9387-5519d9858bc2\") " pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.764373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.764059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d14846-8078-4eed-972f-75e7fff7d7cb-agent-certs\") pod \"konnectivity-agent-w9rjm\" (UID: \"d5d14846-8078-4eed-972f-75e7fff7d7cb\") " pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:31.764937 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.764547    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:44:31.764937 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.764695    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:44:31.764937 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.764777    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pjxsf\""
Sep 16 18:44:31.764937 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.764906    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:44:31.792762 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.792739    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fxgzv"
Sep 16 18:44:31.804694 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.804661    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fxgzv"
Sep 16 18:44:31.816943 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.816890    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal" event={"ID":"62aae6dbe28b0afbce02d1018e37007d","Type":"ContainerStarted","Data":"8b9e14acd2e86a812ce40cbb6b9526a59ab6fcac6f92186a49c6be724c6e0054"}
Sep 16 18:44:31.821669 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.820539    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal" event={"ID":"8611bbd0ccde75189058c67a0a76ec36","Type":"ContainerStarted","Data":"1debf864179de2cc678d3bafc1db82dceb253550035c7f77ef7633e9e112847b"}
Sep 16 18:44:31.847925 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.847897    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:44:31.864586 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:31.864742 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8t2n\" (UniqueName: \"kubernetes.io/projected/79908e31-ecc6-4f92-9af8-bd1ed3d16596-kube-api-access-f8t2n\") pod \"iptables-alerter-k8glm\" (UID: \"79908e31-ecc6-4f92-9af8-bd1ed3d16596\") " pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.864742 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864632    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-cnibin\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.864742 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864660    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-cnibin\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.864742 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79908e31-ecc6-4f92-9af8-bd1ed3d16596-host-slash\") pod \"iptables-alerter-k8glm\" (UID: \"79908e31-ecc6-4f92-9af8-bd1ed3d16596\") " pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.864742 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864710    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-etc-kubernetes\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.864742 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864734    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91214ab9-3932-439b-821e-5746a112e114-dbus\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864758    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-sys\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6764d66c-f9f6-4990-b83c-85216eaae28d-ovnkube-config\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864807    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6764d66c-f9f6-4990-b83c-85216eaae28d-ovn-node-metrics-cert\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864834    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg26z\" (UniqueName: \"kubernetes.io/projected/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-kube-api-access-xg26z\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-cni-binary-copy\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864880    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-systemd-units\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864905    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-var-lib-openvswitch\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-run-ovn-kubernetes\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864954    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.864980    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-socket-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865007    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-sys-fs\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:31.865066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-kubernetes\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865080    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-sysctl-conf\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865105    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-kubelet\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865129    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-etc-selinux\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865155    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91214ab9-3932-439b-821e-5746a112e114-kubelet-config\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865182    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-socket-dir-parent\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-var-lib-cni-multus\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865234    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c17acc3f-e899-405c-aa12-55aa19fc7b86-tmp\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-daemon-config\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865288    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc2d6470-4c6a-4a11-9387-5519d9858bc2-host\") pod \"node-ca-dlqjl\" (UID: \"dc2d6470-4c6a-4a11-9387-5519d9858bc2\") " pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865312    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-tuned\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-run-openvswitch\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-cni-bin\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865408    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79908e31-ecc6-4f92-9af8-bd1ed3d16596-iptables-alerter-script\") pod \"iptables-alerter-k8glm\" (UID: \"79908e31-ecc6-4f92-9af8-bd1ed3d16596\") " pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865436    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-run-k8s-cni-cncf-io\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865435    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-run-ovn-kubernetes\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-var-lib-cni-bin\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.865648 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-conf-dir\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865518    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-sys\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865528    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z967m\" (UniqueName: \"kubernetes.io/projected/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-kube-api-access-z967m\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2chb\" (UniqueName: \"kubernetes.io/projected/c17acc3f-e899-405c-aa12-55aa19fc7b86-kube-api-access-n2chb\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865558    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91214ab9-3932-439b-821e-5746a112e114-dbus\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-log-socket\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865630    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77743e7d-141f-40d6-9fa8-c58c087a0a6a-hosts-file\") pod \"node-resolver-hh97b\" (UID: \"77743e7d-141f-40d6-9fa8-c58c087a0a6a\") " pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-host\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-etc-openvswitch\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865725    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6764d66c-f9f6-4990-b83c-85216eaae28d-env-overrides\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.865790    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.865868    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret podName:91214ab9-3932-439b-821e-5746a112e114 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.365833989 +0000 UTC m=+3.032773120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret") pod "global-pull-secret-syncer-6cn4c" (UID: "91214ab9-3932-439b-821e-5746a112e114") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-kubernetes\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866138    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-kubelet\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-cnibin\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866184    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6764d66c-f9f6-4990-b83c-85216eaae28d-ovnkube-config\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.866569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866202    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91214ab9-3932-439b-821e-5746a112e114-kubelet-config\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866254    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-socket-dir-parent\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866277    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-sysctl-conf\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-var-lib-cni-multus\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866374    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-etc-kubernetes\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866418    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-log-socket\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866468    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77743e7d-141f-40d6-9fa8-c58c087a0a6a-hosts-file\") pod \"node-resolver-hh97b\" (UID: \"77743e7d-141f-40d6-9fa8-c58c087a0a6a\") " pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-host\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866566    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-etc-openvswitch\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866572    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-daemon-config\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866872    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc2d6470-4c6a-4a11-9387-5519d9858bc2-host\") pod \"node-ca-dlqjl\" (UID: \"dc2d6470-4c6a-4a11-9387-5519d9858bc2\") " pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.866958    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6764d66c-f9f6-4990-b83c-85216eaae28d-env-overrides\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867037    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-systemd-units\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867087    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-var-lib-openvswitch\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-var-lib-cni-bin\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867177    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-run-openvswitch\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.867468 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867221    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-cni-bin\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867267    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-run-k8s-cni-cncf-io\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867296    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-conf-dir\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.865791    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7n7j\" (UniqueName: \"kubernetes.io/projected/77743e7d-141f-40d6-9fa8-c58c087a0a6a-kube-api-access-k7n7j\") pod \"node-resolver-hh97b\" (UID: \"77743e7d-141f-40d6-9fa8-c58c087a0a6a\") " pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867324    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-var-lib-kubelet\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-node-log\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867443    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-hostroot\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867480    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-var-lib-kubelet\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867498    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc2d6470-4c6a-4a11-9387-5519d9858bc2-serviceca\") pod \"node-ca-dlqjl\" (UID: \"dc2d6470-4c6a-4a11-9387-5519d9858bc2\") " pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.867559    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.867632    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs podName:1f8f4d51-92a4-4bbe-9201-1ec009f2381b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.367615146 +0000 UTC m=+3.034554290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs") pod "network-metrics-daemon-lcxfx" (UID: "1f8f4d51-92a4-4bbe-9201-1ec009f2381b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867623    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vtx\" (UniqueName: \"kubernetes.io/projected/6764d66c-f9f6-4990-b83c-85216eaae28d-kube-api-access-x2vtx\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867677    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-hostroot\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867681    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-cni-binary-copy\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867751    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tk4j\" (UniqueName: \"kubernetes.io/projected/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-kube-api-access-7tk4j\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.868410 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867790    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqpdb\" (UniqueName: \"kubernetes.io/projected/024fcfb8-4121-4002-96e9-1ba187c9e349-kube-api-access-jqpdb\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77743e7d-141f-40d6-9fa8-c58c087a0a6a-tmp-dir\") pod \"node-resolver-hh97b\" (UID: \"77743e7d-141f-40d6-9fa8-c58c087a0a6a\") " pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.867841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-cni-dir\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868110    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc2d6470-4c6a-4a11-9387-5519d9858bc2-serviceca\") pod \"node-ca-dlqjl\" (UID: \"dc2d6470-4c6a-4a11-9387-5519d9858bc2\") " pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868156    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77743e7d-141f-40d6-9fa8-c58c087a0a6a-tmp-dir\") pod \"node-resolver-hh97b\" (UID: \"77743e7d-141f-40d6-9fa8-c58c087a0a6a\") " pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-node-log\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-modprobe-d\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868251    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-systemd\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868262    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-cni-binary-copy\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-cni-netd\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-cni-netd\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868338    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-device-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868405    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-multus-cni-dir\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868421    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-modprobe-d\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868441    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868469    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-systemd\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868509    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-run-systemd\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-system-cni-dir\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.869297 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868577    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpfjz\" (UniqueName: \"kubernetes.io/projected/dc2d6470-4c6a-4a11-9387-5519d9858bc2-kube-api-access-jpfjz\") pod \"node-ca-dlqjl\" (UID: \"dc2d6470-4c6a-4a11-9387-5519d9858bc2\") " pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868596    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-run-systemd\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868581    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-system-cni-dir\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d14846-8078-4eed-972f-75e7fff7d7cb-agent-certs\") pod \"konnectivity-agent-w9rjm\" (UID: \"d5d14846-8078-4eed-972f-75e7fff7d7cb\") " pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868782    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-run-netns\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-system-cni-dir\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-run-netns\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.868999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869032    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-registration-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869069    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-var-lib-kubelet\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869098    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-run-multus-certs\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d14846-8078-4eed-972f-75e7fff7d7cb-konnectivity-ca\") pod \"konnectivity-agent-w9rjm\" (UID: \"d5d14846-8078-4eed-972f-75e7fff7d7cb\") " pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869143    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-sysctl-d\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869178    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-run-multus-certs\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869200    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-var-lib-kubelet\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869219    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-run\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-lib-modules\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.870283 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-slash\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6764d66c-f9f6-4990-b83c-85216eaae28d-ovnkube-script-lib\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-run\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869337    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-run-netns\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869346    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-host-slash\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-sysconfig\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869388    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-run-ovn\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869424    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-lib-modules\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869311    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-sysctl-d\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869469    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6764d66c-f9f6-4990-b83c-85216eaae28d-run-ovn\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-sysconfig\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869511    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-host-run-netns\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5d14846-8078-4eed-972f-75e7fff7d7cb-konnectivity-ca\") pod \"konnectivity-agent-w9rjm\" (UID: \"d5d14846-8078-4eed-972f-75e7fff7d7cb\") " pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.869853    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6764d66c-f9f6-4990-b83c-85216eaae28d-ovnkube-script-lib\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.871167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.871159    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c17acc3f-e899-405c-aa12-55aa19fc7b86-etc-tuned\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.871998 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.871933    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c17acc3f-e899-405c-aa12-55aa19fc7b86-tmp\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.872369 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.872326    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6764d66c-f9f6-4990-b83c-85216eaae28d-ovn-node-metrics-cert\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.874212 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.874190    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5d14846-8078-4eed-972f-75e7fff7d7cb-agent-certs\") pod \"konnectivity-agent-w9rjm\" (UID: \"d5d14846-8078-4eed-972f-75e7fff7d7cb\") " pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:31.876650 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.876629    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7n7j\" (UniqueName: \"kubernetes.io/projected/77743e7d-141f-40d6-9fa8-c58c087a0a6a-kube-api-access-k7n7j\") pod \"node-resolver-hh97b\" (UID: \"77743e7d-141f-40d6-9fa8-c58c087a0a6a\") " pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:31.877431 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.877400    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg26z\" (UniqueName: \"kubernetes.io/projected/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-kube-api-access-xg26z\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:31.878189 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.878164    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2chb\" (UniqueName: \"kubernetes.io/projected/c17acc3f-e899-405c-aa12-55aa19fc7b86-kube-api-access-n2chb\") pod \"tuned-g2rsd\" (UID: \"c17acc3f-e899-405c-aa12-55aa19fc7b86\") " pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:31.879129 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.879106    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vtx\" (UniqueName: \"kubernetes.io/projected/6764d66c-f9f6-4990-b83c-85216eaae28d-kube-api-access-x2vtx\") pod \"ovnkube-node-lpxk7\" (UID: \"6764d66c-f9f6-4990-b83c-85216eaae28d\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:31.880157 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.880133    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpfjz\" (UniqueName: \"kubernetes.io/projected/dc2d6470-4c6a-4a11-9387-5519d9858bc2-kube-api-access-jpfjz\") pod \"node-ca-dlqjl\" (UID: \"dc2d6470-4c6a-4a11-9387-5519d9858bc2\") " pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:31.880613 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.880589    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z967m\" (UniqueName: \"kubernetes.io/projected/5f1b6ade-4ed0-4051-8c26-a90a089db3ca-kube-api-access-z967m\") pod \"multus-wh7hw\" (UID: \"5f1b6ade-4ed0-4051-8c26-a90a089db3ca\") " pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:31.970469 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:31.970469 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970426    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8t2n\" (UniqueName: \"kubernetes.io/projected/79908e31-ecc6-4f92-9af8-bd1ed3d16596-kube-api-access-f8t2n\") pod \"iptables-alerter-k8glm\" (UID: \"79908e31-ecc6-4f92-9af8-bd1ed3d16596\") " pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.970469 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-cnibin\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.970750 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970514    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-cnibin\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.970750 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970630    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79908e31-ecc6-4f92-9af8-bd1ed3d16596-host-slash\") pod \"iptables-alerter-k8glm\" (UID: \"79908e31-ecc6-4f92-9af8-bd1ed3d16596\") " pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.970750 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970678    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-socket-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.970750 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970704    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-sys-fs\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.970750 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970731    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79908e31-ecc6-4f92-9af8-bd1ed3d16596-host-slash\") pod \"iptables-alerter-k8glm\" (UID: \"79908e31-ecc6-4f92-9af8-bd1ed3d16596\") " pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.970750 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970747    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-etc-selinux\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79908e31-ecc6-4f92-9af8-bd1ed3d16596-iptables-alerter-script\") pod \"iptables-alerter-k8glm\" (UID: \"79908e31-ecc6-4f92-9af8-bd1ed3d16596\") " pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970793    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-sys-fs\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970886    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-socket-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970918    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-etc-selinux\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-cni-binary-copy\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.970987    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tk4j\" (UniqueName: \"kubernetes.io/projected/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-kube-api-access-7tk4j\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971011    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971016    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqpdb\" (UniqueName: \"kubernetes.io/projected/024fcfb8-4121-4002-96e9-1ba187c9e349-kube-api-access-jqpdb\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971067    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-device-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971092    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-system-cni-dir\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971186    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-registration-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-registration-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-system-cni-dir\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971428    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79908e31-ecc6-4f92-9af8-bd1ed3d16596-iptables-alerter-script\") pod \"iptables-alerter-k8glm\" (UID: \"79908e31-ecc6-4f92-9af8-bd1ed3d16596\") " pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.971584 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971499    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/024fcfb8-4121-4002-96e9-1ba187c9e349-device-dir\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.972055 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971595    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-cni-binary-copy\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.972055 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971594    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.972055 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.971850    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.978178 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.978136    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:31.978178 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.978164    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:31.978178 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.978180    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lrx5n for pod openshift-network-diagnostics/network-check-target-cvrx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:31.978455 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:31.978337    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n podName:c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.478294797 +0000 UTC m=+3.145233948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lrx5n" (UniqueName: "kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n") pod "network-check-target-cvrx8" (UID: "c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:31.981822 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.981794    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tk4j\" (UniqueName: \"kubernetes.io/projected/bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226-kube-api-access-7tk4j\") pod \"multus-additional-cni-plugins-hj6rj\" (UID: \"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226\") " pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:31.982461 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.982324    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8t2n\" (UniqueName: \"kubernetes.io/projected/79908e31-ecc6-4f92-9af8-bd1ed3d16596-kube-api-access-f8t2n\") pod \"iptables-alerter-k8glm\" (UID: \"79908e31-ecc6-4f92-9af8-bd1ed3d16596\") " pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:31.982461 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.982418    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqpdb\" (UniqueName: \"kubernetes.io/projected/024fcfb8-4121-4002-96e9-1ba187c9e349-kube-api-access-jqpdb\") pod \"aws-ebs-csi-driver-node-bpflc\" (UID: \"024fcfb8-4121-4002-96e9-1ba187c9e349\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:31.989649 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:31.989631    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:32.006070 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.006042    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:32.044282 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.044248    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dlqjl"
Sep 16 18:44:32.051608 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.051588    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g2rsd"
Sep 16 18:44:32.063595 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.063561    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:32.069400 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.069377    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:32.078350 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.078324    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hh97b"
Sep 16 18:44:32.088318 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.088299    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wh7hw"
Sep 16 18:44:32.104252 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.104224    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hj6rj"
Sep 16 18:44:32.114192 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.114167    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8glm"
Sep 16 18:44:32.121210 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.121188    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc"
Sep 16 18:44:32.375020 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.374943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:32.375020 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.375013    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:32.375227 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:32.375102    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:32.375227 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:32.375163    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs podName:1f8f4d51-92a4-4bbe-9201-1ec009f2381b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.375148201 +0000 UTC m=+4.042087332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs") pod "network-metrics-daemon-lcxfx" (UID: "1f8f4d51-92a4-4bbe-9201-1ec009f2381b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:32.375227 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:32.375102    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:32.375391 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:32.375243    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret podName:91214ab9-3932-439b-821e-5746a112e114 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.37522829 +0000 UTC m=+4.042167430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret") pod "global-pull-secret-syncer-6cn4c" (UID: "91214ab9-3932-439b-821e-5746a112e114") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:32.576591 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.576560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:32.576760 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:32.576736    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:32.576760 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:32.576759    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:32.576831 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:32.576769    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lrx5n for pod openshift-network-diagnostics/network-check-target-cvrx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:32.576831 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:32.576824    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n podName:c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:33.576807767 +0000 UTC m=+4.243746892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrx5n" (UniqueName: "kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n") pod "network-check-target-cvrx8" (UID: "c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:32.808746 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.808642    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:31 +0000 UTC" deadline="2028-03-22 02:38:33.654497517 +0000 UTC"
Sep 16 18:44:32.808746 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:32.808694    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13255h54m0.845806196s"
Sep 16 18:44:33.001523 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:33.001487    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2d6470_4c6a_4a11_9387_5519d9858bc2.slice/crio-f5d2c9fa4a57e36c71178cd6552e25216bfa95a8a69f104e2bfe6e0011fffa2f WatchSource:0}: Error finding container f5d2c9fa4a57e36c71178cd6552e25216bfa95a8a69f104e2bfe6e0011fffa2f: Status 404 returned error can't find the container with id f5d2c9fa4a57e36c71178cd6552e25216bfa95a8a69f104e2bfe6e0011fffa2f
Sep 16 18:44:33.003629 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:33.003596    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79908e31_ecc6_4f92_9af8_bd1ed3d16596.slice/crio-245b646f2f103b863ecade402c0dadb58baccc87593ebf837a310e71fc6e1c14 WatchSource:0}: Error finding container 245b646f2f103b863ecade402c0dadb58baccc87593ebf837a310e71fc6e1c14: Status 404 returned error can't find the container with id 245b646f2f103b863ecade402c0dadb58baccc87593ebf837a310e71fc6e1c14
Sep 16 18:44:33.008290 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:33.008258    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1b6ade_4ed0_4051_8c26_a90a089db3ca.slice/crio-3ce6f437506bed7ee80362c57a37e08666e9d52f4ae2883c08630e9f44427a7f WatchSource:0}: Error finding container 3ce6f437506bed7ee80362c57a37e08666e9d52f4ae2883c08630e9f44427a7f: Status 404 returned error can't find the container with id 3ce6f437506bed7ee80362c57a37e08666e9d52f4ae2883c08630e9f44427a7f
Sep 16 18:44:33.009246 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:33.009221    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77743e7d_141f_40d6_9fa8_c58c087a0a6a.slice/crio-13febefb54e8712c8d8a37354494009d459c1b411a00bc5fa7a06d3b1d365880 WatchSource:0}: Error finding container 13febefb54e8712c8d8a37354494009d459c1b411a00bc5fa7a06d3b1d365880: Status 404 returned error can't find the container with id 13febefb54e8712c8d8a37354494009d459c1b411a00bc5fa7a06d3b1d365880
Sep 16 18:44:33.011579 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:33.011048    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6764d66c_f9f6_4990_b83c_85216eaae28d.slice/crio-699782c34ecb50747c0371ff91a954c13d3ca143b9059701202eccc54197fe1b WatchSource:0}: Error finding container 699782c34ecb50747c0371ff91a954c13d3ca143b9059701202eccc54197fe1b: Status 404 returned error can't find the container with id 699782c34ecb50747c0371ff91a954c13d3ca143b9059701202eccc54197fe1b
Sep 16 18:44:33.012907 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:33.012718    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024fcfb8_4121_4002_96e9_1ba187c9e349.slice/crio-549ab96f1d4590eee2b0877c62f764d018d1eea7df39c58e5d81de2015578eae WatchSource:0}: Error finding container 549ab96f1d4590eee2b0877c62f764d018d1eea7df39c58e5d81de2015578eae: Status 404 returned error can't find the container with id 549ab96f1d4590eee2b0877c62f764d018d1eea7df39c58e5d81de2015578eae
Sep 16 18:44:33.013909 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:33.013821    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0c97eb_ddf7_4495_b8bc_02f2eaa8a226.slice/crio-24f7c03e805f6bfdf30b57cd1544005c2f47caf2ee09f8ab4734663e721ed78b WatchSource:0}: Error finding container 24f7c03e805f6bfdf30b57cd1544005c2f47caf2ee09f8ab4734663e721ed78b: Status 404 returned error can't find the container with id 24f7c03e805f6bfdf30b57cd1544005c2f47caf2ee09f8ab4734663e721ed78b
Sep 16 18:44:33.014778 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:33.014753    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17acc3f_e899_405c_aa12_55aa19fc7b86.slice/crio-860d60ad9b9b614d093a96936d894eefdb6a2562c1f5cf70fe40b3ab43205dc3 WatchSource:0}: Error finding container 860d60ad9b9b614d093a96936d894eefdb6a2562c1f5cf70fe40b3ab43205dc3: Status 404 returned error can't find the container with id 860d60ad9b9b614d093a96936d894eefdb6a2562c1f5cf70fe40b3ab43205dc3
Sep 16 18:44:33.016202 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:44:33.016160    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d14846_8078_4eed_972f_75e7fff7d7cb.slice/crio-b008537886cc7c4d13f100459b724e8d55307bf789621c38b5389a9b808aa177 WatchSource:0}: Error finding container b008537886cc7c4d13f100459b724e8d55307bf789621c38b5389a9b808aa177: Status 404 returned error can't find the container with id b008537886cc7c4d13f100459b724e8d55307bf789621c38b5389a9b808aa177
Sep 16 18:44:33.381138 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.380400    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:33.381138 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.380694    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:33.381138 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.380550    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:33.381138 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.380780    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret podName:91214ab9-3932-439b-821e-5746a112e114 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:35.380759571 +0000 UTC m=+6.047698721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret") pod "global-pull-secret-syncer-6cn4c" (UID: "91214ab9-3932-439b-821e-5746a112e114") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:33.381138 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.380826    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:33.381138 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.380865    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs podName:1f8f4d51-92a4-4bbe-9201-1ec009f2381b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:35.380851306 +0000 UTC m=+6.047790437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs") pod "network-metrics-daemon-lcxfx" (UID: "1f8f4d51-92a4-4bbe-9201-1ec009f2381b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:33.583289 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.582491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:33.583289 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.582728    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:33.583289 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.582755    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:33.583289 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.582773    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lrx5n for pod openshift-network-diagnostics/network-check-target-cvrx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:33.583289 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.582846    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n podName:c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:35.58282539 +0000 UTC m=+6.249764543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrx5n" (UniqueName: "kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n") pod "network-check-target-cvrx8" (UID: "c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:33.815353 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.814524    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:33.815353 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.814686    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:33.815353 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.815116    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:33.815353 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.815202    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:33.815353 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.815273    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:31 +0000 UTC" deadline="2028-03-24 03:46:25.85496021 +0000 UTC"
Sep 16 18:44:33.815353 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.815294    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13305h1m52.039670859s"
Sep 16 18:44:33.816638 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.816408    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:33.816638 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:33.816517    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:33.831464 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.831241    2584 generic.go:356] "Generic (PLEG): container finished" podID="62aae6dbe28b0afbce02d1018e37007d" containerID="dab6fbed84962047fe9a20f19544392bc6335bc764b89ec3ca8a0cb1c4535419" exitCode=0
Sep 16 18:44:33.831464 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.831409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal" event={"ID":"62aae6dbe28b0afbce02d1018e37007d","Type":"ContainerDied","Data":"dab6fbed84962047fe9a20f19544392bc6335bc764b89ec3ca8a0cb1c4535419"}
Sep 16 18:44:33.833816 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.833390    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g2rsd" event={"ID":"c17acc3f-e899-405c-aa12-55aa19fc7b86","Type":"ContainerStarted","Data":"860d60ad9b9b614d093a96936d894eefdb6a2562c1f5cf70fe40b3ab43205dc3"}
Sep 16 18:44:33.840043 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.840019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" event={"ID":"6764d66c-f9f6-4990-b83c-85216eaae28d","Type":"ContainerStarted","Data":"699782c34ecb50747c0371ff91a954c13d3ca143b9059701202eccc54197fe1b"}
Sep 16 18:44:33.845652 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.845586    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hh97b" event={"ID":"77743e7d-141f-40d6-9fa8-c58c087a0a6a","Type":"ContainerStarted","Data":"13febefb54e8712c8d8a37354494009d459c1b411a00bc5fa7a06d3b1d365880"}
Sep 16 18:44:33.863597 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.863563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wh7hw" event={"ID":"5f1b6ade-4ed0-4051-8c26-a90a089db3ca","Type":"ContainerStarted","Data":"3ce6f437506bed7ee80362c57a37e08666e9d52f4ae2883c08630e9f44427a7f"}
Sep 16 18:44:33.869474 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.869404    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8glm" event={"ID":"79908e31-ecc6-4f92-9af8-bd1ed3d16596","Type":"ContainerStarted","Data":"245b646f2f103b863ecade402c0dadb58baccc87593ebf837a310e71fc6e1c14"}
Sep 16 18:44:33.874336 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.874214    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dlqjl" event={"ID":"dc2d6470-4c6a-4a11-9387-5519d9858bc2","Type":"ContainerStarted","Data":"f5d2c9fa4a57e36c71178cd6552e25216bfa95a8a69f104e2bfe6e0011fffa2f"}
Sep 16 18:44:33.879898 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.879831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w9rjm" event={"ID":"d5d14846-8078-4eed-972f-75e7fff7d7cb","Type":"ContainerStarted","Data":"b008537886cc7c4d13f100459b724e8d55307bf789621c38b5389a9b808aa177"}
Sep 16 18:44:33.883828 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.883762    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj6rj" event={"ID":"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226","Type":"ContainerStarted","Data":"24f7c03e805f6bfdf30b57cd1544005c2f47caf2ee09f8ab4734663e721ed78b"}
Sep 16 18:44:33.890238 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.890157    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc" event={"ID":"024fcfb8-4121-4002-96e9-1ba187c9e349","Type":"ContainerStarted","Data":"549ab96f1d4590eee2b0877c62f764d018d1eea7df39c58e5d81de2015578eae"}
Sep 16 18:44:33.922503 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.921840    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal" event={"ID":"8611bbd0ccde75189058c67a0a76ec36","Type":"ContainerStarted","Data":"c55c00d46c0d418866874bebc7defbd2c2bcf9d49fc85db8a20adf83a855e220"}
Sep 16 18:44:33.941668 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:33.941321    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-120.ec2.internal" podStartSLOduration=2.941300048 podStartE2EDuration="2.941300048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:33.940315101 +0000 UTC m=+4.607254278" watchObservedRunningTime="2026-09-16 18:44:33.941300048 +0000 UTC m=+4.608239200"
Sep 16 18:44:34.944476 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:34.944398    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal" event={"ID":"62aae6dbe28b0afbce02d1018e37007d","Type":"ContainerStarted","Data":"ca2dee4b0515287a2eef08948a7bc397c547321be9bbf23a879662c50c563598"}
Sep 16 18:44:35.399917 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:35.399830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:35.399917 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:35.399901    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:35.400145 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.400033    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:35.400145 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.400093    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs podName:1f8f4d51-92a4-4bbe-9201-1ec009f2381b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.400076238 +0000 UTC m=+10.067015365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs") pod "network-metrics-daemon-lcxfx" (UID: "1f8f4d51-92a4-4bbe-9201-1ec009f2381b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:35.400562 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.400473    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:35.400562 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.400522    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret podName:91214ab9-3932-439b-821e-5746a112e114 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.400507257 +0000 UTC m=+10.067446387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret") pod "global-pull-secret-syncer-6cn4c" (UID: "91214ab9-3932-439b-821e-5746a112e114") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:35.602115 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:35.601477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:35.602115 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.601683    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:35.602115 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.601703    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:35.602115 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.601720    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lrx5n for pod openshift-network-diagnostics/network-check-target-cvrx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:35.602115 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.601778    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n podName:c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.601759917 +0000 UTC m=+10.268699045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrx5n" (UniqueName: "kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n") pod "network-check-target-cvrx8" (UID: "c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:35.811898 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:35.811379    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:35.811898 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:35.811387    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:35.811898 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.811510    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:35.811898 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:35.811555    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:35.811898 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.811654    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:35.811898 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:35.811750    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:37.812791 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:37.812691    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:37.813221 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:37.812829    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:37.813221 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:37.813179    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:37.813355 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:37.813296    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:37.813355 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:37.813340    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:37.813479 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:37.813408    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:39.434853 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:39.434815    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:39.435364 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:39.434886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:39.435364 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.435000    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:39.435364 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.435060    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs podName:1f8f4d51-92a4-4bbe-9201-1ec009f2381b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:47.435041522 +0000 UTC m=+18.101980669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs") pod "network-metrics-daemon-lcxfx" (UID: "1f8f4d51-92a4-4bbe-9201-1ec009f2381b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:39.435364 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.435000    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:39.435364 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.435121    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret podName:91214ab9-3932-439b-821e-5746a112e114 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:47.435108392 +0000 UTC m=+18.102047522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret") pod "global-pull-secret-syncer-6cn4c" (UID: "91214ab9-3932-439b-821e-5746a112e114") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:39.636838 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:39.636730    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:39.637015 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.636887    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:39.637015 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.636914    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:39.637015 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.636928    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lrx5n for pod openshift-network-diagnostics/network-check-target-cvrx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:39.637015 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.636991    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n podName:c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:47.636970602 +0000 UTC m=+18.303909730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrx5n" (UniqueName: "kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n") pod "network-check-target-cvrx8" (UID: "c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:39.812161 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:39.812089    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:39.812319 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.812216    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:39.812585 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:39.812567    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:39.812671 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.812657    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:39.812730 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:39.812713    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:39.812811 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:39.812782    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:41.811102 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:41.811058    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:41.811600 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:41.811192    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:41.811782 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:41.811756    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:41.811961 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:41.811933    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:41.812289 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:41.812253    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:41.812439 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:41.812417    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:43.810463 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:43.810430    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:43.810892 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:43.810430    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:43.810892 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:43.810551    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:43.810892 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:43.810443    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:43.810892 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:43.810639    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:43.810892 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:43.810700    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:45.810376 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:45.810333    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:45.810804 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:45.810347    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:45.810804 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:45.810475    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:45.810804 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:45.810469    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:45.810804 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:45.810593    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:45.810804 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:45.810666    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:47.494119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:47.494081    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:47.494604 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:47.494134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:47.494604 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.494256    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:47.494604 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.494256    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:47.494604 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.494325    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs podName:1f8f4d51-92a4-4bbe-9201-1ec009f2381b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.494306895 +0000 UTC m=+34.161246028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs") pod "network-metrics-daemon-lcxfx" (UID: "1f8f4d51-92a4-4bbe-9201-1ec009f2381b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:47.494604 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.494364    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret podName:91214ab9-3932-439b-821e-5746a112e114 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.494344211 +0000 UTC m=+34.161283353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret") pod "global-pull-secret-syncer-6cn4c" (UID: "91214ab9-3932-439b-821e-5746a112e114") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:47.696007 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:47.695967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:47.696196 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.696132    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:47.696196 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.696157    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:47.696196 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.696170    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lrx5n for pod openshift-network-diagnostics/network-check-target-cvrx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:47.696337 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.696226    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n podName:c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.69621143 +0000 UTC m=+34.363150577 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrx5n" (UniqueName: "kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n") pod "network-check-target-cvrx8" (UID: "c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:47.811285 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:47.811181    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:47.811285 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:47.811182    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:47.811489 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:47.811181    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:47.811489 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.811380    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:47.811586 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.811512    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:47.811586 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:47.811578    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:49.811028 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:49.810988    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:49.811476 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:49.811132    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:49.811476 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:49.811211    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:49.811476 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:49.811343    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:49.811476 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:49.811389    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:49.811476 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:49.811456    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:50.977681 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.977510    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g2rsd" event={"ID":"c17acc3f-e899-405c-aa12-55aa19fc7b86","Type":"ContainerStarted","Data":"5d65061ae1673620bd275c371b66285c0375ae8356b25b69611b9a47c7134473"}
Sep 16 18:44:50.979534 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.979377    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" event={"ID":"6764d66c-f9f6-4990-b83c-85216eaae28d","Type":"ContainerStarted","Data":"785cefbf792edf5bee8b12c6c955d5b450ac3e89da9befc082e038f487ee5b49"}
Sep 16 18:44:50.979654 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.979591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" event={"ID":"6764d66c-f9f6-4990-b83c-85216eaae28d","Type":"ContainerStarted","Data":"2447e591cddb51a6dd420f5d8cf34e9b79acb196edfa7acff8bfb226d8a70ad2"}
Sep 16 18:44:50.980954 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.980915    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hh97b" event={"ID":"77743e7d-141f-40d6-9fa8-c58c087a0a6a","Type":"ContainerStarted","Data":"f82dcc373c1907059ad056ae1605b979fd50fa13211a8fe92ee6a5da33862528"}
Sep 16 18:44:50.982538 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.982518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wh7hw" event={"ID":"5f1b6ade-4ed0-4051-8c26-a90a089db3ca","Type":"ContainerStarted","Data":"016351bc00737ed449bd52bc58fc7a5893240f40e8899bce81e75b0109b6cac3"}
Sep 16 18:44:50.984037 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.984012    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dlqjl" event={"ID":"dc2d6470-4c6a-4a11-9387-5519d9858bc2","Type":"ContainerStarted","Data":"779ed9c9606902d3cebc01cfbce844ced0eff45f68d7425eb88940cb5bf315a0"}
Sep 16 18:44:50.985557 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.985532    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w9rjm" event={"ID":"d5d14846-8078-4eed-972f-75e7fff7d7cb","Type":"ContainerStarted","Data":"33c4d31b885b9a19f810b2af744d3e65a1a49dd5a31812841c807e40172b1589"}
Sep 16 18:44:50.987354 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.987330    2584 generic.go:356] "Generic (PLEG): container finished" podID="bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226" containerID="de0ad87de0de63c25abebee7ba25fcead1aee5134d8af92212d463feb7dfc453" exitCode=0
Sep 16 18:44:50.987448 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.987404    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj6rj" event={"ID":"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226","Type":"ContainerDied","Data":"de0ad87de0de63c25abebee7ba25fcead1aee5134d8af92212d463feb7dfc453"}
Sep 16 18:44:50.989180 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:50.989147    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc" event={"ID":"024fcfb8-4121-4002-96e9-1ba187c9e349","Type":"ContainerStarted","Data":"b6581fecfe0bdfb56e81713d3afa54d8e0739ba9c0cfbd1d0c4ebe7426a45ae2"}
Sep 16 18:44:51.001490 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.001452    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g2rsd" podStartSLOduration=3.349522044 podStartE2EDuration="21.001439706s" totalImagesPullingTime="17.651917662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.001079303 +0000 UTC m=+21.668018475" watchObservedRunningTime="2026-09-16 18:44:51.001439706 +0000 UTC m=+21.668378852"
Sep 16 18:44:51.001586 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.001567    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-120.ec2.internal" podStartSLOduration=20.001562077 podStartE2EDuration="20.001562077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:34.961478042 +0000 UTC m=+5.628417189" watchObservedRunningTime="2026-09-16 18:44:51.001562077 +0000 UTC m=+21.668501224"
Sep 16 18:44:51.016615 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.016577    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hh97b" podStartSLOduration=3.357874663 podStartE2EDuration="21.016563448s" totalImagesPullingTime="17.658688785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.016161444 +0000 UTC m=+21.683100589" watchObservedRunningTime="2026-09-16 18:44:51.016563448 +0000 UTC m=+21.683502594"
Sep 16 18:44:51.035108 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.035071    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wh7hw" podStartSLOduration=3.317285801 podStartE2EDuration="21.035055091s" totalImagesPullingTime="17.71776929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.034729103 +0000 UTC m=+21.701668249" watchObservedRunningTime="2026-09-16 18:44:51.035055091 +0000 UTC m=+21.701994237"
Sep 16 18:44:51.075010 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.074969    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dlqjl" podStartSLOduration=3.40827022 podStartE2EDuration="21.074954797s" totalImagesPullingTime="17.666684577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.074540527 +0000 UTC m=+21.741479702" watchObservedRunningTime="2026-09-16 18:44:51.074954797 +0000 UTC m=+21.741893943"
Sep 16 18:44:51.100954 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.100919    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w9rjm" podStartSLOduration=7.467280622 podStartE2EDuration="21.100904814s" totalImagesPullingTime="13.633624192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.100472531 +0000 UTC m=+21.767411697" watchObservedRunningTime="2026-09-16 18:44:51.100904814 +0000 UTC m=+21.767843959"
Sep 16 18:44:51.813778 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.813742    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:51.813778 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.813766    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:51.813976 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.813767    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:51.813976 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:51.813851    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:51.814064 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:51.813968    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:51.814064 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:51.814052    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:51.994485 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.994460    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" event={"ID":"6764d66c-f9f6-4990-b83c-85216eaae28d","Type":"ContainerStarted","Data":"2d6e62597ab73379e19059527c41b36354a4c249ebaf99d9a5e2ac698c71518d"}
Sep 16 18:44:51.994845 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.994542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" event={"ID":"6764d66c-f9f6-4990-b83c-85216eaae28d","Type":"ContainerStarted","Data":"6134f867fe4d4cdba81dec8c06bf8a0a709ed2e88b77278ecd66c269244bd6c3"}
Sep 16 18:44:51.994845 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.994574    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" event={"ID":"6764d66c-f9f6-4990-b83c-85216eaae28d","Type":"ContainerStarted","Data":"316e1f977a420b29e64880d038e38a539795b0fed4270f2f92177a43d3de501e"}
Sep 16 18:44:51.994845 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:51.994609    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" event={"ID":"6764d66c-f9f6-4990-b83c-85216eaae28d","Type":"ContainerStarted","Data":"a4cbd56a37cb86696591c92662ad2902cc4b671a8cec335730e0495d36f5d918"}
Sep 16 18:44:52.137898 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:52.137850    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:44:52.809287 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:52.809171    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:44:52.137869096Z","UUID":"76d005c4-202c-477c-ad6f-0aae407e079f","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:44:52.810813 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:52.810792    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:44:52.810944 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:52.810823    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:44:52.998955 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:52.998904    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8glm" event={"ID":"79908e31-ecc6-4f92-9af8-bd1ed3d16596","Type":"ContainerStarted","Data":"bb7e9a513da4ca82ce9fabe3c41f144fed2d74d3134df2e09d3324a9a753cd42"}
Sep 16 18:44:53.001037 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:53.001010    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc" event={"ID":"024fcfb8-4121-4002-96e9-1ba187c9e349","Type":"ContainerStarted","Data":"9eafe9a2c8fc8200070e4d66bfc79bed27139b40a7f4cfb808b89c99371928e2"}
Sep 16 18:44:53.014931 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:53.014892    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k8glm" podStartSLOduration=5.350396604 podStartE2EDuration="23.014878852s" totalImagesPullingTime="17.664482248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:53.01431095 +0000 UTC m=+23.681250123" watchObservedRunningTime="2026-09-16 18:44:53.014878852 +0000 UTC m=+23.681817999"
Sep 16 18:44:53.810994 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:53.810920    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:53.811146 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:53.810923    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:53.811146 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:53.811053    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:53.811146 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:53.810922    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:53.811304 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:53.811140    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:53.811304 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:53.811228    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:55.626184 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:55.625928    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:55.626622 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:55.626567    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:55.810538 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:55.810511    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:55.810538 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:55.810543    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:55.810770 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:55.810511    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:55.810770 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:55.810604    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:55.810770 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:55.810655    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:55.810770 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:55.810744    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:56.009071 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:56.008958    2584 generic.go:356] "Generic (PLEG): container finished" podID="bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226" containerID="4a71f0b2af3a26caaca76d8abaacb9dadd8dff3126a523fd08fb216a122bd74c" exitCode=0
Sep 16 18:44:56.009071 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:56.009042    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj6rj" event={"ID":"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226","Type":"ContainerDied","Data":"4a71f0b2af3a26caaca76d8abaacb9dadd8dff3126a523fd08fb216a122bd74c"}
Sep 16 18:44:56.015026 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:56.014981    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc" event={"ID":"024fcfb8-4121-4002-96e9-1ba187c9e349","Type":"ContainerStarted","Data":"f2f5300c264c9131d54c133597544ac0fe91e5c4de9b6df9659043d3b82f57e9"}
Sep 16 18:44:56.018233 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:56.018212    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" event={"ID":"6764d66c-f9f6-4990-b83c-85216eaae28d","Type":"ContainerStarted","Data":"e661f24c08150fa809ffed0b7c6a1eb0a6fb4a7a81df2d737ec6d4c6166342ea"}
Sep 16 18:44:56.018330 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:56.018287    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:56.018770 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:56.018756    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w9rjm"
Sep 16 18:44:56.054119 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:56.054064    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bpflc" podStartSLOduration=4.060996854 podStartE2EDuration="26.054048972s" totalImagesPullingTime="21.993052118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:56.053413423 +0000 UTC m=+26.720352569" watchObservedRunningTime="2026-09-16 18:44:56.054048972 +0000 UTC m=+26.720988119"
Sep 16 18:44:57.022657 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:57.022565    2584 generic.go:356] "Generic (PLEG): container finished" podID="bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226" containerID="d2ca61160f13d19d8bc7d50815dbbc9406471286d69f762f5c163b12372b8b17" exitCode=0
Sep 16 18:44:57.022657 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:57.022623    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj6rj" event={"ID":"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226","Type":"ContainerDied","Data":"d2ca61160f13d19d8bc7d50815dbbc9406471286d69f762f5c163b12372b8b17"}
Sep 16 18:44:57.810874 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:57.810798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:57.811034 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:57.810798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:57.811034 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:57.810910    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:57.811034 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:57.810998    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:57.811034 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:57.810798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:57.811242 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:57.811073    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:44:58.028825 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:58.028787    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" event={"ID":"6764d66c-f9f6-4990-b83c-85216eaae28d","Type":"ContainerStarted","Data":"32e47e37c017800427cb683041546d5a3ae604d0b240df7ff916a87c57b66504"}
Sep 16 18:44:58.058634 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:58.058587    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7" podStartSLOduration=10.352344413 podStartE2EDuration="28.058572062s" totalImagesPullingTime="17.706227649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:58.058408121 +0000 UTC m=+28.725347286" watchObservedRunningTime="2026-09-16 18:44:58.058572062 +0000 UTC m=+28.725511208"
Sep 16 18:44:59.033744 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.033543    2584 generic.go:356] "Generic (PLEG): container finished" podID="bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226" containerID="7fac6ede0ab757fb8a1381051a19b458eb6b02fd883d3fd7291f81a6cfbf8239" exitCode=0
Sep 16 18:44:59.034177 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.033618    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj6rj" event={"ID":"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226","Type":"ContainerDied","Data":"7fac6ede0ab757fb8a1381051a19b458eb6b02fd883d3fd7291f81a6cfbf8239"}
Sep 16 18:44:59.034309 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.034233    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:59.034722 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.034322    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:59.034722 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.034366    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:59.049897 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.049870    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:59.050039 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.049971    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:44:59.481659 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.481615    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cvrx8"]
Sep 16 18:44:59.481821 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.481768    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:44:59.481909 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:59.481880    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:44:59.485914 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.485856    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lcxfx"]
Sep 16 18:44:59.486067 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.486053    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:44:59.486190 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:59.486166    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:44:59.486745 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.486652    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6cn4c"]
Sep 16 18:44:59.486844 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:44:59.486749    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:44:59.486844 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:44:59.486835    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:45:00.810883 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:00.810843    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:45:00.811380 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:00.810985    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:45:01.810409 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:01.810375    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:45:01.810583 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:01.810513    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cvrx8" podUID="c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123"
Sep 16 18:45:01.810583 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:01.810570    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:45:01.810716 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:01.810693    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6cn4c" podUID="91214ab9-3932-439b-821e-5746a112e114"
Sep 16 18:45:02.810223 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:02.810189    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:45:02.810691 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:02.810338    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcxfx" podUID="1f8f4d51-92a4-4bbe-9201-1ec009f2381b"
Sep 16 18:45:03.508917 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.508736    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:45:03.509008 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.508923    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:03.509008 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.508967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:45:03.509008 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.509004    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs podName:1f8f4d51-92a4-4bbe-9201-1ec009f2381b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.508982313 +0000 UTC m=+66.175921442 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs") pod "network-metrics-daemon-lcxfx" (UID: "1f8f4d51-92a4-4bbe-9201-1ec009f2381b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:03.509127 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.509054    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:03.509127 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.509099    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret podName:91214ab9-3932-439b-821e-5746a112e114 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.509084716 +0000 UTC m=+66.176023840 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret") pod "global-pull-secret-syncer-6cn4c" (UID: "91214ab9-3932-439b-821e-5746a112e114") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:03.624167 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.624143    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-120.ec2.internal" event="NodeReady"
Sep 16 18:45:03.624340 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.624288    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:45:03.673036 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.672997    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w9zrj"]
Sep 16 18:45:03.684238 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.684191    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rvnq9"]
Sep 16 18:45:03.684464 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.684350    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.687321 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.687296    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:45:03.687440 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.687412    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:45:03.687514 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.687490    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vj57\""
Sep 16 18:45:03.694968 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.694945    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvnq9"]
Sep 16 18:45:03.695073 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.694975    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w9zrj"]
Sep 16 18:45:03.695073 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.695046    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvnq9"
Sep 16 18:45:03.697501 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.697483    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:45:03.697623 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.697517    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:45:03.697623 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.697545    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2bmd5\""
Sep 16 18:45:03.697753 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.697680    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:45:03.710810 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.710783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:45:03.710981 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.710937    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:03.710981 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.710958    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:03.710981 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.710967    2584 projected.go:196] Error preparing data for projected volume kube-api-access-lrx5n for pod openshift-network-diagnostics/network-check-target-cvrx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:03.711199 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.711029    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n podName:c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.71101193 +0000 UTC m=+66.377951060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrx5n" (UniqueName: "kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n") pod "network-check-target-cvrx8" (UID: "c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:03.810608 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.810570    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:45:03.811052 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.810570    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:45:03.811133 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.811114    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-config-volume\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.811183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.811147    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.811236 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.811181    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb7z2\" (UniqueName: \"kubernetes.io/projected/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-kube-api-access-fb7z2\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.811312 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.811257    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-tmp-dir\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.811380 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.811327    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed420e76-3215-428e-aa07-27c0047a9619-cert\") pod \"ingress-canary-rvnq9\" (UID: \"ed420e76-3215-428e-aa07-27c0047a9619\") " pod="openshift-ingress-canary/ingress-canary-rvnq9"
Sep 16 18:45:03.811430 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.811390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qvcc\" (UniqueName: \"kubernetes.io/projected/ed420e76-3215-428e-aa07-27c0047a9619-kube-api-access-9qvcc\") pod \"ingress-canary-rvnq9\" (UID: \"ed420e76-3215-428e-aa07-27c0047a9619\") " pod="openshift-ingress-canary/ingress-canary-rvnq9"
Sep 16 18:45:03.813561 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.813542    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:03.813709 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.813544    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:03.813709 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.813637    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:03.813709 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.813652    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2qmgm\""
Sep 16 18:45:03.912171 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.912139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-tmp-dir\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.912394 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.912192    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed420e76-3215-428e-aa07-27c0047a9619-cert\") pod \"ingress-canary-rvnq9\" (UID: \"ed420e76-3215-428e-aa07-27c0047a9619\") " pod="openshift-ingress-canary/ingress-canary-rvnq9"
Sep 16 18:45:03.912394 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.912225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qvcc\" (UniqueName: \"kubernetes.io/projected/ed420e76-3215-428e-aa07-27c0047a9619-kube-api-access-9qvcc\") pod \"ingress-canary-rvnq9\" (UID: \"ed420e76-3215-428e-aa07-27c0047a9619\") " pod="openshift-ingress-canary/ingress-canary-rvnq9"
Sep 16 18:45:03.912394 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.912242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-config-volume\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.912394 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.912256    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.912394 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.912292    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb7z2\" (UniqueName: \"kubernetes.io/projected/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-kube-api-access-fb7z2\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.912635 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.912395    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:03.912635 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:03.912465    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls podName:5d19db54-ba2c-4ea5-8134-a08ea264e3a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:04.41244316 +0000 UTC m=+35.079382308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls") pod "dns-default-w9zrj" (UID: "5d19db54-ba2c-4ea5-8134-a08ea264e3a7") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:03.912635 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.912498    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-tmp-dir\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.912921 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.912904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-config-volume\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.916677 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.916655    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed420e76-3215-428e-aa07-27c0047a9619-cert\") pod \"ingress-canary-rvnq9\" (UID: \"ed420e76-3215-428e-aa07-27c0047a9619\") " pod="openshift-ingress-canary/ingress-canary-rvnq9"
Sep 16 18:45:03.926745 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.926715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb7z2\" (UniqueName: \"kubernetes.io/projected/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-kube-api-access-fb7z2\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:03.926882 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:03.926783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qvcc\" (UniqueName: \"kubernetes.io/projected/ed420e76-3215-428e-aa07-27c0047a9619-kube-api-access-9qvcc\") pod \"ingress-canary-rvnq9\" (UID: \"ed420e76-3215-428e-aa07-27c0047a9619\") " pod="openshift-ingress-canary/ingress-canary-rvnq9"
Sep 16 18:45:04.002879 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:04.002786    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvnq9"
Sep 16 18:45:04.051692 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:04.051664    2584 generic.go:356] "Generic (PLEG): container finished" podID="bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226" containerID="29926584bb4ea04cceafa0d0dc8590ec585a03679d9298ef0f31262c9b42b0c1" exitCode=0
Sep 16 18:45:04.051889 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:04.051726    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj6rj" event={"ID":"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226","Type":"ContainerDied","Data":"29926584bb4ea04cceafa0d0dc8590ec585a03679d9298ef0f31262c9b42b0c1"}
Sep 16 18:45:04.312473 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:04.312386    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvnq9"]
Sep 16 18:45:04.317178 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:45:04.317152    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded420e76_3215_428e_aa07_27c0047a9619.slice/crio-d2b708a3fec25c9f5b3fc1ef4ce2b4a750df46402126833ee876da185dc64ce1 WatchSource:0}: Error finding container d2b708a3fec25c9f5b3fc1ef4ce2b4a750df46402126833ee876da185dc64ce1: Status 404 returned error can't find the container with id d2b708a3fec25c9f5b3fc1ef4ce2b4a750df46402126833ee876da185dc64ce1
Sep 16 18:45:04.415895 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:04.415855    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:04.416081 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:04.415979    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:04.416081 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:04.416043    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls podName:5d19db54-ba2c-4ea5-8134-a08ea264e3a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:05.416027257 +0000 UTC m=+36.082966381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls") pod "dns-default-w9zrj" (UID: "5d19db54-ba2c-4ea5-8134-a08ea264e3a7") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:04.810700 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:04.810667    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:45:04.818260 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:04.818233    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:04.818389 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:04.818280    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rlx6q\""
Sep 16 18:45:05.058057 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:05.058021    2584 generic.go:356] "Generic (PLEG): container finished" podID="bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226" containerID="bb3f5b10acc3f9a5ae672786c7b30f5dff53764f33bab73175e494bf6cda20f9" exitCode=0
Sep 16 18:45:05.058240 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:05.058099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj6rj" event={"ID":"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226","Type":"ContainerDied","Data":"bb3f5b10acc3f9a5ae672786c7b30f5dff53764f33bab73175e494bf6cda20f9"}
Sep 16 18:45:05.059507 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:05.059483    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvnq9" event={"ID":"ed420e76-3215-428e-aa07-27c0047a9619","Type":"ContainerStarted","Data":"d2b708a3fec25c9f5b3fc1ef4ce2b4a750df46402126833ee876da185dc64ce1"}
Sep 16 18:45:05.424419 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:05.424389    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:05.424655 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:05.424538    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:05.424655 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:05.424603    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls podName:5d19db54-ba2c-4ea5-8134-a08ea264e3a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.424586201 +0000 UTC m=+38.091525334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls") pod "dns-default-w9zrj" (UID: "5d19db54-ba2c-4ea5-8134-a08ea264e3a7") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:06.065915 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:06.065880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj6rj" event={"ID":"bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226","Type":"ContainerStarted","Data":"0fb797fd111f0854fc24e51d64feff804c7d290543264fbfc47cde97c51b3f2e"}
Sep 16 18:45:06.093386 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:06.093335    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hj6rj" podStartSLOduration=8.163437801 podStartE2EDuration="36.093318695s" totalImagesPullingTime="27.929880894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:06.091617911 +0000 UTC m=+36.758557057" watchObservedRunningTime="2026-09-16 18:45:06.093318695 +0000 UTC m=+36.760257841"
Sep 16 18:45:07.069880 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:07.069849    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvnq9" event={"ID":"ed420e76-3215-428e-aa07-27c0047a9619","Type":"ContainerStarted","Data":"c9164b05b0440e1ed67621a65f10da9956df1207da840142efd91aefb67ca74d"}
Sep 16 18:45:07.088382 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:07.088290    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rvnq9" podStartSLOduration=1.570849226 podStartE2EDuration="4.088260401s" totalImagesPullingTime="2.517411175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:07.087503741 +0000 UTC m=+37.754442897" watchObservedRunningTime="2026-09-16 18:45:07.088260401 +0000 UTC m=+37.755199547"
Sep 16 18:45:07.439693 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:07.439660    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:07.439834 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:07.439802    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:07.439882 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:07.439870    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls podName:5d19db54-ba2c-4ea5-8134-a08ea264e3a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:11.439853412 +0000 UTC m=+42.106792543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls") pod "dns-default-w9zrj" (UID: "5d19db54-ba2c-4ea5-8134-a08ea264e3a7") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:11.466549 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:11.466508    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:11.466949 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:11.466664    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:11.466949 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:11.466729    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls podName:5d19db54-ba2c-4ea5-8134-a08ea264e3a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:19.466714123 +0000 UTC m=+50.133653253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls") pod "dns-default-w9zrj" (UID: "5d19db54-ba2c-4ea5-8134-a08ea264e3a7") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:19.522019 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:19.521981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:19.522431 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:19.522134    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:19.522431 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:19.522198    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls podName:5d19db54-ba2c-4ea5-8134-a08ea264e3a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.522182577 +0000 UTC m=+66.189121702 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls") pod "dns-default-w9zrj" (UID: "5d19db54-ba2c-4ea5-8134-a08ea264e3a7") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:31.049286 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:31.049245    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpxk7"
Sep 16 18:45:35.537506 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.537467    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:45:35.537907 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.537517    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:45:35.537907 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.537653    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:45:35.537907 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:35.537767    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:35.537907 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:35.537862    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls podName:5d19db54-ba2c-4ea5-8134-a08ea264e3a7 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:07.537805194 +0000 UTC m=+98.204744329 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls") pod "dns-default-w9zrj" (UID: "5d19db54-ba2c-4ea5-8134-a08ea264e3a7") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:35.540485 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.540455    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:35.540485 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.540486    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:35.547759 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:35.547741    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:45:35.547872 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:35.547790    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs podName:1f8f4d51-92a4-4bbe-9201-1ec009f2381b nodeName:}" failed. No retries permitted until 2026-09-16 18:46:39.547776111 +0000 UTC m=+130.214715240 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs") pod "network-metrics-daemon-lcxfx" (UID: "1f8f4d51-92a4-4bbe-9201-1ec009f2381b") : secret "metrics-daemon-secret" not found
Sep 16 18:45:35.550776 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.550759    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91214ab9-3932-439b-821e-5746a112e114-original-pull-secret\") pod \"global-pull-secret-syncer-6cn4c\" (UID: \"91214ab9-3932-439b-821e-5746a112e114\") " pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:45:35.624957 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.624929    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6cn4c"
Sep 16 18:45:35.738610 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.738581    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:45:35.742057 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.742034    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:35.752719 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.752699    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:35.762421 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.762401    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrx5n\" (UniqueName: \"kubernetes.io/projected/c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123-kube-api-access-lrx5n\") pod \"network-check-target-cvrx8\" (UID: \"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123\") " pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:45:35.916278 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.916236    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6cn4c"]
Sep 16 18:45:35.921442 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:45:35.921409    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91214ab9_3932_439b_821e_5746a112e114.slice/crio-2cc7180730f054f7f9ad6f36b397542243c469357057a254861e2936d2299d05 WatchSource:0}: Error finding container 2cc7180730f054f7f9ad6f36b397542243c469357057a254861e2936d2299d05: Status 404 returned error can't find the container with id 2cc7180730f054f7f9ad6f36b397542243c469357057a254861e2936d2299d05
Sep 16 18:45:35.924751 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.924731    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2qmgm\""
Sep 16 18:45:35.932518 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:35.932500    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:45:36.134535 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:36.134501    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6cn4c" event={"ID":"91214ab9-3932-439b-821e-5746a112e114","Type":"ContainerStarted","Data":"2cc7180730f054f7f9ad6f36b397542243c469357057a254861e2936d2299d05"}
Sep 16 18:45:36.202643 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:36.202578    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cvrx8"]
Sep 16 18:45:36.205375 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:45:36.205353    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f93bb3_6b63_4d1e_bf7e_46e0b3e93123.slice/crio-26ed453530b602bf49f86a9caea4329efe7f075a34c2ec39ca0c0db3254fd10a WatchSource:0}: Error finding container 26ed453530b602bf49f86a9caea4329efe7f075a34c2ec39ca0c0db3254fd10a: Status 404 returned error can't find the container with id 26ed453530b602bf49f86a9caea4329efe7f075a34c2ec39ca0c0db3254fd10a
Sep 16 18:45:37.138367 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:37.138327    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cvrx8" event={"ID":"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123","Type":"ContainerStarted","Data":"26ed453530b602bf49f86a9caea4329efe7f075a34c2ec39ca0c0db3254fd10a"}
Sep 16 18:45:41.150936 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:41.150899    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6cn4c" event={"ID":"91214ab9-3932-439b-821e-5746a112e114","Type":"ContainerStarted","Data":"1110e448a9954a8fe19ab5150b82461da2f1921d4bc601ea02efb22e639e4603"}
Sep 16 18:45:41.152500 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:41.152476    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cvrx8" event={"ID":"c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123","Type":"ContainerStarted","Data":"3105471caedffffaf2a697c620341ce5270a8b1ec331efd850509b1d6ab07a8f"}
Sep 16 18:45:41.152630 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:41.152614    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:45:41.172508 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:41.172465    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6cn4c" podStartSLOduration=66.609388749 podStartE2EDuration="1m11.172453189s" totalImagesPullingTime="4.56306444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:41.171219577 +0000 UTC m=+71.838158724" watchObservedRunningTime="2026-09-16 18:45:41.172453189 +0000 UTC m=+71.839392335"
Sep 16 18:45:41.191174 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:41.191138    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cvrx8" podStartSLOduration=66.91558865 podStartE2EDuration="1m11.191126823s" totalImagesPullingTime="4.275538173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:41.19024704 +0000 UTC m=+71.857186206" watchObservedRunningTime="2026-09-16 18:45:41.191126823 +0000 UTC m=+71.858065969"
Sep 16 18:45:43.293070 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.293036    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz"]
Sep 16 18:45:43.296067 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.296050    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz"
Sep 16 18:45:43.299560 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.299536    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.299755 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.299647    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.300656 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.300641    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jmdjr\""
Sep 16 18:45:43.304655 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.304636    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz"]
Sep 16 18:45:43.381354 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.381329    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6gwh\" (UniqueName: \"kubernetes.io/projected/16d42993-1963-4a2b-851b-cd2bd9dfb477-kube-api-access-b6gwh\") pod \"volume-data-source-validator-86695b7cc7-n26xz\" (UID: \"16d42993-1963-4a2b-851b-cd2bd9dfb477\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz"
Sep 16 18:45:43.407065 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.407014    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-h97rl"]
Sep 16 18:45:43.410066 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.410049    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-646544f686-74gxd"]
Sep 16 18:45:43.410179 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.410162    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.412977 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.412948    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:45:43.413072 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.412991    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.413072 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.412997    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.413072 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.413029    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.415518 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.415502    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:45:43.416355 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.416333    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:45:43.416447 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.416359    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j7v7s\""
Sep 16 18:45:43.416614 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.416339    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:45:43.416808 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.416765    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fp478\""
Sep 16 18:45:43.416912 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.416893    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.416975 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.416904    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:45:43.417062 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.416905    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.417609 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.417592    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:45:43.422861 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.422842    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:45:43.437362 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.437345    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-h97rl"]
Sep 16 18:45:43.438340 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.438321    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-646544f686-74gxd"]
Sep 16 18:45:43.482404 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482386    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-stats-auth\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.482489 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482414    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3674164a-4141-4d00-ba6a-45fff63302fb-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.482489 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482432    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.482489 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482457    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3674164a-4141-4d00-ba6a-45fff63302fb-tmp\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.482593 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482514    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3674164a-4141-4d00-ba6a-45fff63302fb-snapshots\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.482593 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6gwh\" (UniqueName: \"kubernetes.io/projected/16d42993-1963-4a2b-851b-cd2bd9dfb477-kube-api-access-b6gwh\") pod \"volume-data-source-validator-86695b7cc7-n26xz\" (UID: \"16d42993-1963-4a2b-851b-cd2bd9dfb477\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz"
Sep 16 18:45:43.482593 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482576    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3674164a-4141-4d00-ba6a-45fff63302fb-serving-cert\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.482714 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482592    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp86k\" (UniqueName: \"kubernetes.io/projected/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-kube-api-access-hp86k\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.482714 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw5wf\" (UniqueName: \"kubernetes.io/projected/3674164a-4141-4d00-ba6a-45fff63302fb-kube-api-access-lw5wf\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.482714 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482632    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-default-certificate\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.482714 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482648    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.482714 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.482665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3674164a-4141-4d00-ba6a-45fff63302fb-service-ca-bundle\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.491837 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.491814    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6gwh\" (UniqueName: \"kubernetes.io/projected/16d42993-1963-4a2b-851b-cd2bd9dfb477-kube-api-access-b6gwh\") pod \"volume-data-source-validator-86695b7cc7-n26xz\" (UID: \"16d42993-1963-4a2b-851b-cd2bd9dfb477\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz"
Sep 16 18:45:43.508950 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.508932    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"]
Sep 16 18:45:43.512020 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.512003    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"]
Sep 16 18:45:43.512141 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.512125    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:43.514962 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.514946    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.515030 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.514979    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.515079 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.515063    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pjpps\""
Sep 16 18:45:43.515120 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.515090    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.515373 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.515352    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:45:43.520746 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.520727    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7jjkz\""
Sep 16 18:45:43.520840 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.520809    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:45:43.524662 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.524645    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:45:43.524753 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.524650    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.524753 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.524682    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.531623 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.531601    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"]
Sep 16 18:45:43.532704 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.532686    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"]
Sep 16 18:45:43.583125 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583066    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6fd\" (UniqueName: \"kubernetes.io/projected/6678e175-f02f-43f4-9a15-53efd70313a1-kube-api-access-mg6fd\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xmn8h\" (UID: \"6678e175-f02f-43f4-9a15-53efd70313a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.583125 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583107    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6678e175-f02f-43f4-9a15-53efd70313a1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xmn8h\" (UID: \"6678e175-f02f-43f4-9a15-53efd70313a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.583244 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3674164a-4141-4d00-ba6a-45fff63302fb-serving-cert\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.583244 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583168    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp86k\" (UniqueName: \"kubernetes.io/projected/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-kube-api-access-hp86k\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.583244 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw5wf\" (UniqueName: \"kubernetes.io/projected/3674164a-4141-4d00-ba6a-45fff63302fb-kube-api-access-lw5wf\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.583244 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583221    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-default-certificate\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.583244 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583238    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.583482 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583260    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:43.583482 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583307    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3674164a-4141-4d00-ba6a-45fff63302fb-service-ca-bundle\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.583482 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:43.583401    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:43.584386 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583435    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-stats-auth\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.584386 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3674164a-4141-4d00-ba6a-45fff63302fb-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.584386 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.583943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.584386 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.584007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3674164a-4141-4d00-ba6a-45fff63302fb-tmp\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.584386 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:43.584033    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.084007315 +0000 UTC m=+74.750946439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : secret "router-metrics-certs-default" not found
Sep 16 18:45:43.584386 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.584315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3674164a-4141-4d00-ba6a-45fff63302fb-tmp\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.584386 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.584336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3674164a-4141-4d00-ba6a-45fff63302fb-service-ca-bundle\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.584762 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:43.584416    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.084399354 +0000 UTC m=+74.751338495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:43.584762 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.584490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjqhm\" (UniqueName: \"kubernetes.io/projected/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-kube-api-access-kjqhm\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:43.584762 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.584542    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3674164a-4141-4d00-ba6a-45fff63302fb-snapshots\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.584762 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.584586    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6678e175-f02f-43f4-9a15-53efd70313a1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xmn8h\" (UID: \"6678e175-f02f-43f4-9a15-53efd70313a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.588708 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.585250    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3674164a-4141-4d00-ba6a-45fff63302fb-snapshots\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.588708 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.586811    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3674164a-4141-4d00-ba6a-45fff63302fb-serving-cert\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.588708 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.586897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-default-certificate\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.588708 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.587191    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-stats-auth\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.588708 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.587513    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3674164a-4141-4d00-ba6a-45fff63302fb-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.596341 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.596322    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw5wf\" (UniqueName: \"kubernetes.io/projected/3674164a-4141-4d00-ba6a-45fff63302fb-kube-api-access-lw5wf\") pod \"insights-operator-5844b856db-h97rl\" (UID: \"3674164a-4141-4d00-ba6a-45fff63302fb\") " pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.597692 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.597675    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp86k\" (UniqueName: \"kubernetes.io/projected/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-kube-api-access-hp86k\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:43.604687 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.604670    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz"
Sep 16 18:45:43.685983 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.685954    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjqhm\" (UniqueName: \"kubernetes.io/projected/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-kube-api-access-kjqhm\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:43.686129 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.685991    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6678e175-f02f-43f4-9a15-53efd70313a1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xmn8h\" (UID: \"6678e175-f02f-43f4-9a15-53efd70313a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.686129 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.686011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6fd\" (UniqueName: \"kubernetes.io/projected/6678e175-f02f-43f4-9a15-53efd70313a1-kube-api-access-mg6fd\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xmn8h\" (UID: \"6678e175-f02f-43f4-9a15-53efd70313a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.686129 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.686037    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6678e175-f02f-43f4-9a15-53efd70313a1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xmn8h\" (UID: \"6678e175-f02f-43f4-9a15-53efd70313a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.686318 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.686150    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:43.686318 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:43.686258    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:45:43.686421 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:43.686333    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls podName:b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.18631248 +0000 UTC m=+74.853251618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5lg7r" (UID: "b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa") : secret "samples-operator-tls" not found
Sep 16 18:45:43.686578 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.686559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6678e175-f02f-43f4-9a15-53efd70313a1-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xmn8h\" (UID: \"6678e175-f02f-43f4-9a15-53efd70313a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.688094 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.688078    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6678e175-f02f-43f4-9a15-53efd70313a1-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xmn8h\" (UID: \"6678e175-f02f-43f4-9a15-53efd70313a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.694852 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.694832    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjqhm\" (UniqueName: \"kubernetes.io/projected/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-kube-api-access-kjqhm\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:43.695044 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.695026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6fd\" (UniqueName: \"kubernetes.io/projected/6678e175-f02f-43f4-9a15-53efd70313a1-kube-api-access-mg6fd\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xmn8h\" (UID: \"6678e175-f02f-43f4-9a15-53efd70313a1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.720120 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.720099    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-h97rl"
Sep 16 18:45:43.825673 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.825650    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"
Sep 16 18:45:43.872051 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.872023    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz"]
Sep 16 18:45:43.875561 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:45:43.875533    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d42993_1963_4a2b_851b_cd2bd9dfb477.slice/crio-34a8f7f8b8ef8ccc927fe46ada7da71cda21b18dd37ba1ec4e1aead1e64c089c WatchSource:0}: Error finding container 34a8f7f8b8ef8ccc927fe46ada7da71cda21b18dd37ba1ec4e1aead1e64c089c: Status 404 returned error can't find the container with id 34a8f7f8b8ef8ccc927fe46ada7da71cda21b18dd37ba1ec4e1aead1e64c089c
Sep 16 18:45:43.989182 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:43.989150    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-h97rl"]
Sep 16 18:45:43.993041 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:45:43.993016    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3674164a_4141_4d00_ba6a_45fff63302fb.slice/crio-15687f77c89856f9f3a02f8e09075650c584c5f9a636819f4337dda375a00432 WatchSource:0}: Error finding container 15687f77c89856f9f3a02f8e09075650c584c5f9a636819f4337dda375a00432: Status 404 returned error can't find the container with id 15687f77c89856f9f3a02f8e09075650c584c5f9a636819f4337dda375a00432
Sep 16 18:45:44.089052 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.088976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:44.089052 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.089047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:44.089209 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:44.089128    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:44.089209 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:44.089177    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.089162617 +0000 UTC m=+75.756101745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : secret "router-metrics-certs-default" not found
Sep 16 18:45:44.089209 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:44.089190    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.089184442 +0000 UTC m=+75.756123567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:44.093941 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.093919    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h"]
Sep 16 18:45:44.096246 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:45:44.096220    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6678e175_f02f_43f4_9a15_53efd70313a1.slice/crio-28bd1b8240234090386ded7e56d4056d8e2e8e52598afc697426ce9f895c9fc6 WatchSource:0}: Error finding container 28bd1b8240234090386ded7e56d4056d8e2e8e52598afc697426ce9f895c9fc6: Status 404 returned error can't find the container with id 28bd1b8240234090386ded7e56d4056d8e2e8e52598afc697426ce9f895c9fc6
Sep 16 18:45:44.160778 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.160752    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h" event={"ID":"6678e175-f02f-43f4-9a15-53efd70313a1","Type":"ContainerStarted","Data":"28bd1b8240234090386ded7e56d4056d8e2e8e52598afc697426ce9f895c9fc6"}
Sep 16 18:45:44.161852 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.161831    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz" event={"ID":"16d42993-1963-4a2b-851b-cd2bd9dfb477","Type":"ContainerStarted","Data":"34a8f7f8b8ef8ccc927fe46ada7da71cda21b18dd37ba1ec4e1aead1e64c089c"}
Sep 16 18:45:44.162948 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.162931    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h97rl" event={"ID":"3674164a-4141-4d00-ba6a-45fff63302fb","Type":"ContainerStarted","Data":"15687f77c89856f9f3a02f8e09075650c584c5f9a636819f4337dda375a00432"}
Sep 16 18:45:44.189405 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.189384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:44.189523 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:44.189509    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:45:44.189571 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:44.189561    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls podName:b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.189546949 +0000 UTC m=+75.856486073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5lg7r" (UID: "b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa") : secret "samples-operator-tls" not found
Sep 16 18:45:44.344654 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.344575    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q"]
Sep 16 18:45:44.348986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.348968    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q"
Sep 16 18:45:44.352318 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.352301    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xf26p\""
Sep 16 18:45:44.363946 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.363921    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q"]
Sep 16 18:45:44.391059 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.391028    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vtv\" (UniqueName: \"kubernetes.io/projected/098cca9f-23f0-4259-954a-fc6a8dbac714-kube-api-access-89vtv\") pod \"network-check-source-6f8b768455-sbk9q\" (UID: \"098cca9f-23f0-4259-954a-fc6a8dbac714\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q"
Sep 16 18:45:44.492248 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.492203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vtv\" (UniqueName: \"kubernetes.io/projected/098cca9f-23f0-4259-954a-fc6a8dbac714-kube-api-access-89vtv\") pod \"network-check-source-6f8b768455-sbk9q\" (UID: \"098cca9f-23f0-4259-954a-fc6a8dbac714\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q"
Sep 16 18:45:44.501560 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.501533    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vtv\" (UniqueName: \"kubernetes.io/projected/098cca9f-23f0-4259-954a-fc6a8dbac714-kube-api-access-89vtv\") pod \"network-check-source-6f8b768455-sbk9q\" (UID: \"098cca9f-23f0-4259-954a-fc6a8dbac714\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q"
Sep 16 18:45:44.657356 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.657280    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q"
Sep 16 18:45:44.933758 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:44.933719    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q"]
Sep 16 18:45:44.938175 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:45:44.938142    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098cca9f_23f0_4259_954a_fc6a8dbac714.slice/crio-36b722d2b86ee14c3e81bc880bb6008bc662b1ba4ca73c1500ef52be0b81f6e5 WatchSource:0}: Error finding container 36b722d2b86ee14c3e81bc880bb6008bc662b1ba4ca73c1500ef52be0b81f6e5: Status 404 returned error can't find the container with id 36b722d2b86ee14c3e81bc880bb6008bc662b1ba4ca73c1500ef52be0b81f6e5
Sep 16 18:45:45.097873 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:45.097838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:45.098059 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:45.097935    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:45.098120 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:45.098064    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:45.098171 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:45.098127    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.098106677 +0000 UTC m=+77.765045821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : secret "router-metrics-certs-default" not found
Sep 16 18:45:45.098627 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:45.098575    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.098558078 +0000 UTC m=+77.765497207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:45.168798 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:45.168760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q" event={"ID":"098cca9f-23f0-4259-954a-fc6a8dbac714","Type":"ContainerStarted","Data":"b9eedcaef62ff5343eca2ac49e543da70d429cb0e8cfc73dc75e56efe2c4c4b4"}
Sep 16 18:45:45.168985 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:45.168820    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q" event={"ID":"098cca9f-23f0-4259-954a-fc6a8dbac714","Type":"ContainerStarted","Data":"36b722d2b86ee14c3e81bc880bb6008bc662b1ba4ca73c1500ef52be0b81f6e5"}
Sep 16 18:45:45.192248 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:45.191299    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-sbk9q" podStartSLOduration=1.191277313 podStartE2EDuration="1.191277313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:45.189978106 +0000 UTC m=+75.856917252" watchObservedRunningTime="2026-09-16 18:45:45.191277313 +0000 UTC m=+75.858216452"
Sep 16 18:45:45.199229 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:45.198380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:45.199229 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:45.198885    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:45:45.199229 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:45.198943    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls podName:b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.198923155 +0000 UTC m=+77.865862285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5lg7r" (UID: "b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa") : secret "samples-operator-tls" not found
Sep 16 18:45:46.174054 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:46.174019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz" event={"ID":"16d42993-1963-4a2b-851b-cd2bd9dfb477","Type":"ContainerStarted","Data":"154944eae2f16ced48ee27297ee0699d9c962e1894a85a6cc29260780e9f4d92"}
Sep 16 18:45:46.204320 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:46.204245    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n26xz" podStartSLOduration=1.724925629 podStartE2EDuration="3.204226104s" totalImagesPullingTime="1.479300475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:46.201820023 +0000 UTC m=+76.868759192" watchObservedRunningTime="2026-09-16 18:45:46.204226104 +0000 UTC m=+76.871165252"
Sep 16 18:45:47.115415 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:47.115321    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:47.115415 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:47.115392    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:47.115597 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:47.115479    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:47.115597 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:47.115535    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:51.115520041 +0000 UTC m=+81.782459165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:47.115597 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:47.115551    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:51.115545777 +0000 UTC m=+81.782484901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : secret "router-metrics-certs-default" not found
Sep 16 18:45:47.178500 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:47.178457    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h" event={"ID":"6678e175-f02f-43f4-9a15-53efd70313a1","Type":"ContainerStarted","Data":"630b0ef3950363e8a501f5e485ce558187e1e4fae7ffcbcd5ad2aba71c79afe0"}
Sep 16 18:45:47.180093 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:47.180071    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h97rl" event={"ID":"3674164a-4141-4d00-ba6a-45fff63302fb","Type":"ContainerStarted","Data":"efbcc8aab96890765f219d038e05be74e82461ebd54821e603a2b2fb07702f00"}
Sep 16 18:45:47.215499 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:47.215447    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h" podStartSLOduration=1.563649343 podStartE2EDuration="4.21542768s" totalImagesPullingTime="2.651778337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:47.21314899 +0000 UTC m=+77.880088136" watchObservedRunningTime="2026-09-16 18:45:47.21542768 +0000 UTC m=+77.882366822"
Sep 16 18:45:47.215843 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:47.215825    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:47.215976 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:47.215946    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:45:47.216140 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:47.216126    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls podName:b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:51.216106154 +0000 UTC m=+81.883045310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5lg7r" (UID: "b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa") : secret "samples-operator-tls" not found
Sep 16 18:45:47.250736 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:47.250677    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-h97rl" podStartSLOduration=1.498827716 podStartE2EDuration="4.250654745s" totalImagesPullingTime="2.751827029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:47.249461297 +0000 UTC m=+77.916400462" watchObservedRunningTime="2026-09-16 18:45:47.250654745 +0000 UTC m=+77.917593892"
Sep 16 18:45:48.587965 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.587935    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tdz2g"]
Sep 16 18:45:48.591349 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.591327    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.595332 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.595303    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:45:48.595467 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.595391    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:45:48.595467 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.595433    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sppxv\""
Sep 16 18:45:48.612538 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.612516    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdz2g"]
Sep 16 18:45:48.627149 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.627131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.627251 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.627156    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b56eaab7-f00c-48cf-841d-d46f859324e8-data-volume\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.627251 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.627178    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.627251 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.627216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsp9\" (UniqueName: \"kubernetes.io/projected/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-api-access-nhsp9\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.627371 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.627321    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b56eaab7-f00c-48cf-841d-d46f859324e8-crio-socket\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.728162 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.728140    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsp9\" (UniqueName: \"kubernetes.io/projected/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-api-access-nhsp9\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.728304 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.728193    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b56eaab7-f00c-48cf-841d-d46f859324e8-crio-socket\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.728304 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.728242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.728304 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.728284    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b56eaab7-f00c-48cf-841d-d46f859324e8-data-volume\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.728439 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.728313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.728439 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.728353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b56eaab7-f00c-48cf-841d-d46f859324e8-crio-socket\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.728439 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:48.728391    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:48.728439 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:48.728420    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:48.728575 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:48.728475    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.228456253 +0000 UTC m=+79.895395390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:48.728575 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:48.728493    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.228484914 +0000 UTC m=+79.895424038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:48.728650 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.728600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b56eaab7-f00c-48cf-841d-d46f859324e8-data-volume\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:48.737490 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:48.737474    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsp9\" (UniqueName: \"kubernetes.io/projected/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-api-access-nhsp9\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:49.232447 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.232416    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:49.232447 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.232449    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:49.232639 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:49.232539    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:49.232639 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:49.232551    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:49.232639 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:49.232598    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.232581381 +0000 UTC m=+80.899520527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:49.232639 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:49.232613    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.232606647 +0000 UTC m=+80.899545771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:49.291746 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.291720    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"]
Sep 16 18:45:49.294753 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.294739    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:49.297855 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.297831    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ntnn8\""
Sep 16 18:45:49.298133 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.298116    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:45:49.298693 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.298674    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:45:49.310458 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.310435    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"]
Sep 16 18:45:49.332770 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.332736    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:49.332889 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.332871    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:49.433273 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.433246    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:49.433381 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.433305    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:49.433433 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:49.433386    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:49.433464 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:49.433445    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert podName:e13f7d12-6a82-469f-8c78-52ef0e7af2b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.933432028 +0000 UTC m=+80.600371152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2275q" (UID: "e13f7d12-6a82-469f-8c78-52ef0e7af2b3") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:49.433840 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.433820    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:49.936140 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:49.936108    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:49.936520 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:49.936233    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:49.936520 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:49.936302    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert podName:e13f7d12-6a82-469f-8c78-52ef0e7af2b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.936288875 +0000 UTC m=+81.603227998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2275q" (UID: "e13f7d12-6a82-469f-8c78-52ef0e7af2b3") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:50.238876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.238797    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:50.238876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.238841    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:50.239045 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:50.238931    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:50.239045 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:50.238944    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:50.239045 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:50.238989    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.238975485 +0000 UTC m=+82.905914610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:50.239045 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:50.239003    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.238997494 +0000 UTC m=+82.905936618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:50.504091 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.504030    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hh97b_77743e7d-141f-40d6-9fa8-c58c087a0a6a/dns-node-resolver/0.log"
Sep 16 18:45:50.606857 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.606829    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-nhrs6"]
Sep 16 18:45:50.609760 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.609731    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.612485 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.612462    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:45:50.612663 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.612642    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:45:50.613206 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.613190    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:45:50.613687 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.613674    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:45:50.613773 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.613755    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xpw9t\""
Sep 16 18:45:50.621101 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.621084    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-nhrs6"]
Sep 16 18:45:50.642922 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.642901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4aa6770c-fc71-4aed-9150-1c6a3a5074e5-signing-key\") pod \"service-ca-568869fdc5-nhrs6\" (UID: \"4aa6770c-fc71-4aed-9150-1c6a3a5074e5\") " pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.643001 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.642933    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4aa6770c-fc71-4aed-9150-1c6a3a5074e5-signing-cabundle\") pod \"service-ca-568869fdc5-nhrs6\" (UID: \"4aa6770c-fc71-4aed-9150-1c6a3a5074e5\") " pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.643001 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.642961    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pt67\" (UniqueName: \"kubernetes.io/projected/4aa6770c-fc71-4aed-9150-1c6a3a5074e5-kube-api-access-8pt67\") pod \"service-ca-568869fdc5-nhrs6\" (UID: \"4aa6770c-fc71-4aed-9150-1c6a3a5074e5\") " pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.743830 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.743799    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4aa6770c-fc71-4aed-9150-1c6a3a5074e5-signing-key\") pod \"service-ca-568869fdc5-nhrs6\" (UID: \"4aa6770c-fc71-4aed-9150-1c6a3a5074e5\") " pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.743909 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.743840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4aa6770c-fc71-4aed-9150-1c6a3a5074e5-signing-cabundle\") pod \"service-ca-568869fdc5-nhrs6\" (UID: \"4aa6770c-fc71-4aed-9150-1c6a3a5074e5\") " pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.743909 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.743863    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pt67\" (UniqueName: \"kubernetes.io/projected/4aa6770c-fc71-4aed-9150-1c6a3a5074e5-kube-api-access-8pt67\") pod \"service-ca-568869fdc5-nhrs6\" (UID: \"4aa6770c-fc71-4aed-9150-1c6a3a5074e5\") " pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.744387 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.744373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4aa6770c-fc71-4aed-9150-1c6a3a5074e5-signing-cabundle\") pod \"service-ca-568869fdc5-nhrs6\" (UID: \"4aa6770c-fc71-4aed-9150-1c6a3a5074e5\") " pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.746143 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.746126    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4aa6770c-fc71-4aed-9150-1c6a3a5074e5-signing-key\") pod \"service-ca-568869fdc5-nhrs6\" (UID: \"4aa6770c-fc71-4aed-9150-1c6a3a5074e5\") " pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.754515 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.754466    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pt67\" (UniqueName: \"kubernetes.io/projected/4aa6770c-fc71-4aed-9150-1c6a3a5074e5-kube-api-access-8pt67\") pod \"service-ca-568869fdc5-nhrs6\" (UID: \"4aa6770c-fc71-4aed-9150-1c6a3a5074e5\") " pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.918407 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.918385    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-nhrs6"
Sep 16 18:45:50.944663 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:50.944625    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:50.944941 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:50.944772    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:50.944941 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:50.944847    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert podName:e13f7d12-6a82-469f-8c78-52ef0e7af2b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.944831709 +0000 UTC m=+83.611770833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2275q" (UID: "e13f7d12-6a82-469f-8c78-52ef0e7af2b3") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:51.146048 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:51.145980    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:51.146048 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:51.146024    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:51.146201 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:51.146138    2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:45:51.146235 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:51.146215    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.146197915 +0000 UTC m=+89.813137044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : secret "router-metrics-certs-default" not found
Sep 16 18:45:51.146235 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:51.146230    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.146223502 +0000 UTC m=+89.813162627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:51.191614 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:51.191588    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-nhrs6"]
Sep 16 18:45:51.194424 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:45:51.194397    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa6770c_fc71_4aed_9150_1c6a3a5074e5.slice/crio-af59e0bf10a412823e411a1aa020ebca689b27732b99d178819871ea9ca9c11f WatchSource:0}: Error finding container af59e0bf10a412823e411a1aa020ebca689b27732b99d178819871ea9ca9c11f: Status 404 returned error can't find the container with id af59e0bf10a412823e411a1aa020ebca689b27732b99d178819871ea9ca9c11f
Sep 16 18:45:51.247147 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:51.247122    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:51.247285 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:51.247253    2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:45:51.247338 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:51.247322    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls podName:b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.247307787 +0000 UTC m=+89.914246947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5lg7r" (UID: "b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa") : secret "samples-operator-tls" not found
Sep 16 18:45:51.304194 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:51.304175    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dlqjl_dc2d6470-4c6a-4a11-9387-5519d9858bc2/node-ca/0.log"
Sep 16 18:45:52.199136 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:52.199003    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-nhrs6" event={"ID":"4aa6770c-fc71-4aed-9150-1c6a3a5074e5","Type":"ContainerStarted","Data":"af59e0bf10a412823e411a1aa020ebca689b27732b99d178819871ea9ca9c11f"}
Sep 16 18:45:52.255535 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:52.255496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:52.255535 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:52.255541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:52.255761 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:52.255619    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:52.255761 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:52.255656    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:52.255761 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:52.255681    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.255663487 +0000 UTC m=+86.922602613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:52.255761 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:52.255716    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.255698438 +0000 UTC m=+86.922637576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:52.506318 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:52.506230    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rvnq9_ed420e76-3215-428e-aa07-27c0047a9619/serve-healthcheck-canary/0.log"
Sep 16 18:45:52.962820 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:52.962783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:52.962991 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:52.962899    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:52.962991 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:52.962958    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert podName:e13f7d12-6a82-469f-8c78-52ef0e7af2b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.962942649 +0000 UTC m=+87.629881777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2275q" (UID: "e13f7d12-6a82-469f-8c78-52ef0e7af2b3") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:54.209430 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:54.209393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-nhrs6" event={"ID":"4aa6770c-fc71-4aed-9150-1c6a3a5074e5","Type":"ContainerStarted","Data":"ce425ea56560f64cbcab86e2431933dd2fa2af8e70b7227dd3b3d16126ada13c"}
Sep 16 18:45:54.230308 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:54.230246    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-nhrs6" podStartSLOduration=1.889169911 podStartE2EDuration="4.230231936s" totalImagesPullingTime="2.341062025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:54.22877291 +0000 UTC m=+84.895712056" watchObservedRunningTime="2026-09-16 18:45:54.230231936 +0000 UTC m=+84.897171082"
Sep 16 18:45:56.293474 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:56.293438    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:56.293858 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:56.293480    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:45:56.293858 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:56.293587    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:56.293858 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:56.293636    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:56.293858 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:56.293660    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.293640953 +0000 UTC m=+94.960580096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:56.293858 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:56.293677    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.293666211 +0000 UTC m=+94.960605352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:56.998884 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:56.998846    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:45:56.999086 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:56.999023    2584 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:45:56.999146 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:56.999103    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert podName:e13f7d12-6a82-469f-8c78-52ef0e7af2b3 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.999083584 +0000 UTC m=+95.666022712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2275q" (UID: "e13f7d12-6a82-469f-8c78-52ef0e7af2b3") : secret "networking-console-plugin-cert" not found
Sep 16 18:45:59.214747 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:59.214706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:59.215183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:59.214764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:59.215183 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:45:59.214895    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle podName:4d092af1-0b63-4f2c-89f2-6cffbe76bdbd nodeName:}" failed. No retries permitted until 2026-09-16 18:46:15.214877325 +0000 UTC m=+105.881816449 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle") pod "router-default-646544f686-74gxd" (UID: "4d092af1-0b63-4f2c-89f2-6cffbe76bdbd") : configmap references non-existent config key: service-ca.crt
Sep 16 18:45:59.217165 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:59.217143    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-metrics-certs\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:45:59.315200 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:59.315163    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:59.317755 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:59.317733    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5lg7r\" (UID: \"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:59.421216 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:59.421186    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"
Sep 16 18:45:59.722201 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:45:59.722173    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r"]
Sep 16 18:46:00.229936 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:00.229903    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r" event={"ID":"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa","Type":"ContainerStarted","Data":"defbfb7992920530878c30e2b4522b3d4e01a26b57b41c588f725f02e1e0dec2"}
Sep 16 18:46:02.237945 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:02.237907    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r" event={"ID":"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa","Type":"ContainerStarted","Data":"0b7054eb9f1c7c44f96352f29f37301dc2da4822fb3c4587103a1adec3691d36"}
Sep 16 18:46:02.238304 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:02.237976    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r" event={"ID":"b2a8a5e2-a38d-4d89-a85f-a49efdbdb6aa","Type":"ContainerStarted","Data":"d73eca4db4486548f6cce3a0fe0b6402c83c78d8de3a60afaebcdcf289aa4726"}
Sep 16 18:46:02.257619 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:02.257576    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5lg7r" podStartSLOduration=17.610759309 podStartE2EDuration="19.257560453s" totalImagesPullingTime="1.646801144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.256380999 +0000 UTC m=+92.923320166" watchObservedRunningTime="2026-09-16 18:46:02.257560453 +0000 UTC m=+92.924499602"
Sep 16 18:46:04.356244 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:04.356202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:46:04.356650 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:04.356253    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:46:04.356650 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:46:04.356363    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:04.356650 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:46:04.356423    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls podName:b56eaab7-f00c-48cf-841d-d46f859324e8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.356408914 +0000 UTC m=+111.023348038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdz2g" (UID: "b56eaab7-f00c-48cf-841d-d46f859324e8") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:04.356769 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:04.356715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b56eaab7-f00c-48cf-841d-d46f859324e8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:46:05.062040 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:05.062002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:46:05.064443 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:05.064420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e13f7d12-6a82-469f-8c78-52ef0e7af2b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2275q\" (UID: \"e13f7d12-6a82-469f-8c78-52ef0e7af2b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:46:05.205969 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:05.205942    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"
Sep 16 18:46:05.473682 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:05.473654    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q"]
Sep 16 18:46:05.476636 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:05.476604    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13f7d12_6a82_469f_8c78_52ef0e7af2b3.slice/crio-92c256db1441d651075e119a90e81a5a946e14de0e61c1aeaf53edd57cee0e07 WatchSource:0}: Error finding container 92c256db1441d651075e119a90e81a5a946e14de0e61c1aeaf53edd57cee0e07: Status 404 returned error can't find the container with id 92c256db1441d651075e119a90e81a5a946e14de0e61c1aeaf53edd57cee0e07
Sep 16 18:46:06.251490 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:06.251454    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q" event={"ID":"e13f7d12-6a82-469f-8c78-52ef0e7af2b3","Type":"ContainerStarted","Data":"92c256db1441d651075e119a90e81a5a946e14de0e61c1aeaf53edd57cee0e07"}
Sep 16 18:46:07.256045 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:07.256011    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q" event={"ID":"e13f7d12-6a82-469f-8c78-52ef0e7af2b3","Type":"ContainerStarted","Data":"142b7e17ef4faea43b1f7c7747c2bd0f61a13616d9236e9755c2c51c83eee8de"}
Sep 16 18:46:07.274617 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:07.274569    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2275q" podStartSLOduration=17.248262991 podStartE2EDuration="18.274555382s" totalImagesPullingTime="1.026292391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:07.273500418 +0000 UTC m=+97.940439561" watchObservedRunningTime="2026-09-16 18:46:07.274555382 +0000 UTC m=+97.941494529"
Sep 16 18:46:07.578520 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:07.578450    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:46:07.580785 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:07.580759    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d19db54-ba2c-4ea5-8134-a08ea264e3a7-metrics-tls\") pod \"dns-default-w9zrj\" (UID: \"5d19db54-ba2c-4ea5-8134-a08ea264e3a7\") " pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:46:07.597437 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:07.597415    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vj57\""
Sep 16 18:46:07.604994 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:07.604979    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:46:07.879083 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:07.879015    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w9zrj"]
Sep 16 18:46:07.882318 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:07.882288    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d19db54_ba2c_4ea5_8134_a08ea264e3a7.slice/crio-6e9c4a0dc17500763580fb042878669cb9c2ceaf1d141e36dbc632dd3c18c070 WatchSource:0}: Error finding container 6e9c4a0dc17500763580fb042878669cb9c2ceaf1d141e36dbc632dd3c18c070: Status 404 returned error can't find the container with id 6e9c4a0dc17500763580fb042878669cb9c2ceaf1d141e36dbc632dd3c18c070
Sep 16 18:46:08.260151 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:08.260119    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9zrj" event={"ID":"5d19db54-ba2c-4ea5-8134-a08ea264e3a7","Type":"ContainerStarted","Data":"6e9c4a0dc17500763580fb042878669cb9c2ceaf1d141e36dbc632dd3c18c070"}
Sep 16 18:46:10.268734 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:10.268695    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9zrj" event={"ID":"5d19db54-ba2c-4ea5-8134-a08ea264e3a7","Type":"ContainerStarted","Data":"d8de48959ccb81dda776164ceb82ae0b2a211626c2bfd22470ae7190b9c5a251"}
Sep 16 18:46:10.269206 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:10.268760    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w9zrj" event={"ID":"5d19db54-ba2c-4ea5-8134-a08ea264e3a7","Type":"ContainerStarted","Data":"1849da5a04f36cf67ba35dc29ef49cda00269a3b0d49bd4b1d32e175f7e83faa"}
Sep 16 18:46:10.269206 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:10.268801    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:46:10.287632 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:10.287589    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w9zrj" podStartSLOduration=65.751388831 podStartE2EDuration="1m7.287578051s" totalImagesPullingTime="1.53618922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:10.286202464 +0000 UTC m=+100.953141611" watchObservedRunningTime="2026-09-16 18:46:10.287578051 +0000 UTC m=+100.954517194"
Sep 16 18:46:12.157747 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:12.157717    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cvrx8"
Sep 16 18:46:15.238148 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.238112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:46:15.238746 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.238729    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d092af1-0b63-4f2c-89f2-6cffbe76bdbd-service-ca-bundle\") pod \"router-default-646544f686-74gxd\" (UID: \"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd\") " pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:46:15.313200 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.313172    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-549f4b54cd-lvknp"]
Sep 16 18:46:15.369014 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.368983    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549f4b54cd-lvknp"]
Sep 16 18:46:15.369137 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.369070    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.375251 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.375227    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:46:15.375375 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.375253    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:46:15.377166 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.377147    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:46:15.380709 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.380690    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hvbtm\""
Sep 16 18:46:15.381695 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.381675    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:46:15.440025 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.439995    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba382a88-6fe6-42f0-a236-ed3bf71e4078-installation-pull-secrets\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.440175 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.440032    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba382a88-6fe6-42f0-a236-ed3bf71e4078-bound-sa-token\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.440175 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.440096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba382a88-6fe6-42f0-a236-ed3bf71e4078-registry-tls\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.440175 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.440127    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba382a88-6fe6-42f0-a236-ed3bf71e4078-trusted-ca\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.440368 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.440189    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba382a88-6fe6-42f0-a236-ed3bf71e4078-ca-trust-extracted\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.440368 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.440225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba382a88-6fe6-42f0-a236-ed3bf71e4078-registry-certificates\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.440368 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.440283    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpdzm\" (UniqueName: \"kubernetes.io/projected/ba382a88-6fe6-42f0-a236-ed3bf71e4078-kube-api-access-wpdzm\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.440368 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.440320    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba382a88-6fe6-42f0-a236-ed3bf71e4078-image-registry-private-configuration\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.525314 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.525246    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:46:15.541527 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.541503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba382a88-6fe6-42f0-a236-ed3bf71e4078-registry-certificates\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.541643 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.541536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpdzm\" (UniqueName: \"kubernetes.io/projected/ba382a88-6fe6-42f0-a236-ed3bf71e4078-kube-api-access-wpdzm\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.541643 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.541567    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba382a88-6fe6-42f0-a236-ed3bf71e4078-image-registry-private-configuration\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.541643 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.541609    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba382a88-6fe6-42f0-a236-ed3bf71e4078-installation-pull-secrets\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.541643 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.541636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba382a88-6fe6-42f0-a236-ed3bf71e4078-bound-sa-token\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.541853 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.541675    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba382a88-6fe6-42f0-a236-ed3bf71e4078-registry-tls\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.541853 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.541701    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba382a88-6fe6-42f0-a236-ed3bf71e4078-trusted-ca\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.541853 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.541745    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba382a88-6fe6-42f0-a236-ed3bf71e4078-ca-trust-extracted\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.542223 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.542203    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba382a88-6fe6-42f0-a236-ed3bf71e4078-ca-trust-extracted\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.542521 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.542501    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba382a88-6fe6-42f0-a236-ed3bf71e4078-registry-certificates\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.542787 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.542772    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba382a88-6fe6-42f0-a236-ed3bf71e4078-trusted-ca\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.544182 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.544156    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba382a88-6fe6-42f0-a236-ed3bf71e4078-registry-tls\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.555809 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.555783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba382a88-6fe6-42f0-a236-ed3bf71e4078-installation-pull-secrets\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.556034 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.555984    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba382a88-6fe6-42f0-a236-ed3bf71e4078-image-registry-private-configuration\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.556810 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.556786    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba382a88-6fe6-42f0-a236-ed3bf71e4078-bound-sa-token\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.557322 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.557302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpdzm\" (UniqueName: \"kubernetes.io/projected/ba382a88-6fe6-42f0-a236-ed3bf71e4078-kube-api-access-wpdzm\") pod \"image-registry-549f4b54cd-lvknp\" (UID: \"ba382a88-6fe6-42f0-a236-ed3bf71e4078\") " pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.677241 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.677207    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:15.823787 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.823768    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-646544f686-74gxd"]
Sep 16 18:46:15.825474 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:15.825451    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d092af1_0b63_4f2c_89f2_6cffbe76bdbd.slice/crio-c7949f5744403ace33160257ad6358afe66e189b4d40db32edb77c3025ed1b62 WatchSource:0}: Error finding container c7949f5744403ace33160257ad6358afe66e189b4d40db32edb77c3025ed1b62: Status 404 returned error can't find the container with id c7949f5744403ace33160257ad6358afe66e189b4d40db32edb77c3025ed1b62
Sep 16 18:46:15.965820 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:15.965782    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549f4b54cd-lvknp"]
Sep 16 18:46:15.970163 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:15.970100    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba382a88_6fe6_42f0_a236_ed3bf71e4078.slice/crio-cb46f34809d5f07b12fc6e9e4e2541a0bd63004dc90a8e45a9de6fb8e148564c WatchSource:0}: Error finding container cb46f34809d5f07b12fc6e9e4e2541a0bd63004dc90a8e45a9de6fb8e148564c: Status 404 returned error can't find the container with id cb46f34809d5f07b12fc6e9e4e2541a0bd63004dc90a8e45a9de6fb8e148564c
Sep 16 18:46:16.290417 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:16.290383    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549f4b54cd-lvknp" event={"ID":"ba382a88-6fe6-42f0-a236-ed3bf71e4078","Type":"ContainerStarted","Data":"0c3d6c7bcb31b53957429baa62fab3a0d2ff7b18730583ec9d6e0cf1a799588e"}
Sep 16 18:46:16.290845 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:16.290444    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549f4b54cd-lvknp" event={"ID":"ba382a88-6fe6-42f0-a236-ed3bf71e4078","Type":"ContainerStarted","Data":"cb46f34809d5f07b12fc6e9e4e2541a0bd63004dc90a8e45a9de6fb8e148564c"}
Sep 16 18:46:16.290845 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:16.290494    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:16.292131 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:16.292110    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-646544f686-74gxd" event={"ID":"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd","Type":"ContainerStarted","Data":"c9af9a3ff0eb6ff1b193364bf1a8f02456c281825b243d5d15cb304c5c23123f"}
Sep 16 18:46:16.292242 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:16.292149    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-646544f686-74gxd" event={"ID":"4d092af1-0b63-4f2c-89f2-6cffbe76bdbd","Type":"ContainerStarted","Data":"c7949f5744403ace33160257ad6358afe66e189b4d40db32edb77c3025ed1b62"}
Sep 16 18:46:16.315165 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:16.315122    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-549f4b54cd-lvknp" podStartSLOduration=1.315110784 podStartE2EDuration="1.315110784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:16.314173735 +0000 UTC m=+106.981112905" watchObservedRunningTime="2026-09-16 18:46:16.315110784 +0000 UTC m=+106.982049929"
Sep 16 18:46:16.336246 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:16.336205    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-646544f686-74gxd" podStartSLOduration=33.336192769 podStartE2EDuration="33.336192769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:16.334896466 +0000 UTC m=+107.001835627" watchObservedRunningTime="2026-09-16 18:46:16.336192769 +0000 UTC m=+107.003131916"
Sep 16 18:46:16.525444 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:16.525408    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:46:16.528059 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:16.528035    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:46:17.295543 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:17.295502    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:46:17.296651 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:17.296631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-646544f686-74gxd"
Sep 16 18:46:20.275138 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:20.275105    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w9zrj"
Sep 16 18:46:20.379362 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:20.379311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:46:20.383420 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:20.383395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b56eaab7-f00c-48cf-841d-d46f859324e8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdz2g\" (UID: \"b56eaab7-f00c-48cf-841d-d46f859324e8\") " pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:46:20.399618 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:20.399598    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdz2g"
Sep 16 18:46:20.710420 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:20.710385    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdz2g"]
Sep 16 18:46:20.714343 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:20.714312    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56eaab7_f00c_48cf_841d_d46f859324e8.slice/crio-accbffa556cc34d32062eb95bd22b33e44c75a9ca894beb33e18090b2915688d WatchSource:0}: Error finding container accbffa556cc34d32062eb95bd22b33e44c75a9ca894beb33e18090b2915688d: Status 404 returned error can't find the container with id accbffa556cc34d32062eb95bd22b33e44c75a9ca894beb33e18090b2915688d
Sep 16 18:46:21.311945 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:21.311911    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdz2g" event={"ID":"b56eaab7-f00c-48cf-841d-d46f859324e8","Type":"ContainerStarted","Data":"bf72c772e0455116beac6d005341f2e8ce854a54bb5576499c83d0af783c820a"}
Sep 16 18:46:21.312333 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:21.311974    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdz2g" event={"ID":"b56eaab7-f00c-48cf-841d-d46f859324e8","Type":"ContainerStarted","Data":"accbffa556cc34d32062eb95bd22b33e44c75a9ca894beb33e18090b2915688d"}
Sep 16 18:46:22.318042 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.317999    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdz2g" event={"ID":"b56eaab7-f00c-48cf-841d-d46f859324e8","Type":"ContainerStarted","Data":"e8329795ed4b2980f54fda8903e1180ea7f2beb1467060c3c34541ad24e5ef79"}
Sep 16 18:46:22.753687 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.753653    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"]
Sep 16 18:46:22.757986 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.757965    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:22.760834 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.760812    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:46:22.761982 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.761961    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:46:22.762205 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.762189    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:46:22.762312 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.762220    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:46:22.762312 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.762244    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9kjxx\""
Sep 16 18:46:22.762430 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.762382    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:46:22.767982 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.767960    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"]
Sep 16 18:46:22.900159 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.900135    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d4bf852-bbc0-4429-a0e6-c9a723915f11-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:22.900249 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.900201    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxnd\" (UniqueName: \"kubernetes.io/projected/7d4bf852-bbc0-4429-a0e6-c9a723915f11-kube-api-access-pnxnd\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:22.900317 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.900253    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d4bf852-bbc0-4429-a0e6-c9a723915f11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:22.900317 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:22.900288    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d4bf852-bbc0-4429-a0e6-c9a723915f11-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.001074 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.001045    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d4bf852-bbc0-4429-a0e6-c9a723915f11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.001214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.001078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d4bf852-bbc0-4429-a0e6-c9a723915f11-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.001214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.001112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d4bf852-bbc0-4429-a0e6-c9a723915f11-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.001214 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.001149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxnd\" (UniqueName: \"kubernetes.io/projected/7d4bf852-bbc0-4429-a0e6-c9a723915f11-kube-api-access-pnxnd\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.001811 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.001786    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d4bf852-bbc0-4429-a0e6-c9a723915f11-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.003391 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.003368    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d4bf852-bbc0-4429-a0e6-c9a723915f11-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.003445 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.003391    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d4bf852-bbc0-4429-a0e6-c9a723915f11-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.010749 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.010700    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxnd\" (UniqueName: \"kubernetes.io/projected/7d4bf852-bbc0-4429-a0e6-c9a723915f11-kube-api-access-pnxnd\") pod \"prometheus-operator-68bcd7f987-gn8d8\" (UID: \"7d4bf852-bbc0-4429-a0e6-c9a723915f11\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.069238 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.069195    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"
Sep 16 18:46:23.323534 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.323459    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdz2g" event={"ID":"b56eaab7-f00c-48cf-841d-d46f859324e8","Type":"ContainerStarted","Data":"27ecd3d5c0cc16c887196229052f87c5e6eacf00089c7fbaf1e8c3d910180d22"}
Sep 16 18:46:23.340049 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.340014    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8"]
Sep 16 18:46:23.342335 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:23.342314    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4bf852_bbc0_4429_a0e6_c9a723915f11.slice/crio-023c92b72d68a58492d347344f3e72f91fd717fd292ed7bde955646c4b7b28fd WatchSource:0}: Error finding container 023c92b72d68a58492d347344f3e72f91fd717fd292ed7bde955646c4b7b28fd: Status 404 returned error can't find the container with id 023c92b72d68a58492d347344f3e72f91fd717fd292ed7bde955646c4b7b28fd
Sep 16 18:46:23.345913 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:23.345874    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tdz2g" podStartSLOduration=33.341605424 podStartE2EDuration="35.345859711s" totalImagesPullingTime="2.004254287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:23.34478422 +0000 UTC m=+114.011723368" watchObservedRunningTime="2026-09-16 18:46:23.345859711 +0000 UTC m=+114.012798857"
Sep 16 18:46:24.328222 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:24.328192    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8" event={"ID":"7d4bf852-bbc0-4429-a0e6-c9a723915f11","Type":"ContainerStarted","Data":"023c92b72d68a58492d347344f3e72f91fd717fd292ed7bde955646c4b7b28fd"}
Sep 16 18:46:25.334989 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:25.334952    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8" event={"ID":"7d4bf852-bbc0-4429-a0e6-c9a723915f11","Type":"ContainerStarted","Data":"99331e27d1da3c182c52c72705ce292a55eaba3f881aa0895b380698d14c84b6"}
Sep 16 18:46:25.335450 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:25.335031    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8" event={"ID":"7d4bf852-bbc0-4429-a0e6-c9a723915f11","Type":"ContainerStarted","Data":"91c2d6633ab7ae5bbf873dd7154c45f2df7e3d470628441dcb7e406e12d43b5b"}
Sep 16 18:46:25.352867 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:25.352824    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn8d8" podStartSLOduration=2.130817261 podStartE2EDuration="3.352811708s" totalImagesPullingTime="1.221994447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:25.352781502 +0000 UTC m=+116.019720648" watchObservedRunningTime="2026-09-16 18:46:25.352811708 +0000 UTC m=+116.019750854"
Sep 16 18:46:27.105071 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.105036    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"]
Sep 16 18:46:27.108855 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.108833    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.111858 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.111833    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:46:27.111951 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.111833    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:46:27.112184 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.112168    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-txv6f\""
Sep 16 18:46:27.130073 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.130052    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"]
Sep 16 18:46:27.165649 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.165624    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9cbmd"]
Sep 16 18:46:27.168798 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.168780    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.173413 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.173394    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:46:27.173520 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.173477    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:46:27.173733 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.173715    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pzlzb\""
Sep 16 18:46:27.173992 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.173977    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:46:27.237809 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.237786    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5862522-1ff7-47d1-b29d-24a445296331-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.237923 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.237822    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5862522-1ff7-47d1-b29d-24a445296331-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.237923 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.237846    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jkc9\" (UniqueName: \"kubernetes.io/projected/f5862522-1ff7-47d1-b29d-24a445296331-kube-api-access-5jkc9\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.237994 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.237913    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5862522-1ff7-47d1-b29d-24a445296331-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.338482 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5862522-1ff7-47d1-b29d-24a445296331-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.338629 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338494    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-tls\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.338629 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338519    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b04c2df7-04c9-416f-9da1-99d27b8d9142-sys\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.338746 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338653    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b04c2df7-04c9-416f-9da1-99d27b8d9142-root\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.338746 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-textfile\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.338746 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338732    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.338896 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338780    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.338896 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338807    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-wtmp\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.338974 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5862522-1ff7-47d1-b29d-24a445296331-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.338974 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.338922    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24bc\" (UniqueName: \"kubernetes.io/projected/b04c2df7-04c9-416f-9da1-99d27b8d9142-kube-api-access-s24bc\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.339071 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.339057    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5862522-1ff7-47d1-b29d-24a445296331-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.339132 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.339098    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jkc9\" (UniqueName: \"kubernetes.io/projected/f5862522-1ff7-47d1-b29d-24a445296331-kube-api-access-5jkc9\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.339183 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.339127    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04c2df7-04c9-416f-9da1-99d27b8d9142-metrics-client-ca\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.339771 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.339748    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5862522-1ff7-47d1-b29d-24a445296331-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.341356 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.341336    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5862522-1ff7-47d1-b29d-24a445296331-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.341461 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.341444    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5862522-1ff7-47d1-b29d-24a445296331-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.347719 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.347700    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jkc9\" (UniqueName: \"kubernetes.io/projected/f5862522-1ff7-47d1-b29d-24a445296331-kube-api-access-5jkc9\") pod \"openshift-state-metrics-67dcf8c979-lj8dn\" (UID: \"f5862522-1ff7-47d1-b29d-24a445296331\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.420686 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.420666    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"
Sep 16 18:46:27.439657 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b04c2df7-04c9-416f-9da1-99d27b8d9142-root\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.439756 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439664    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-textfile\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.439756 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439690    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.439756 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439723    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.439756 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439752    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-wtmp\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.439942 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439751    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b04c2df7-04c9-416f-9da1-99d27b8d9142-root\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.439942 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439868    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24bc\" (UniqueName: \"kubernetes.io/projected/b04c2df7-04c9-416f-9da1-99d27b8d9142-kube-api-access-s24bc\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.439942 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439879    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-wtmp\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.440096 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04c2df7-04c9-416f-9da1-99d27b8d9142-metrics-client-ca\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.440096 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439984    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-tls\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.440096 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.439986    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-textfile\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.440096 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.440012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b04c2df7-04c9-416f-9da1-99d27b8d9142-sys\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.440096 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.440083    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b04c2df7-04c9-416f-9da1-99d27b8d9142-sys\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.440335 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.440292    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.440505 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.440485    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04c2df7-04c9-416f-9da1-99d27b8d9142-metrics-client-ca\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.442115 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.442086    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-tls\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.442570 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.442549    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b04c2df7-04c9-416f-9da1-99d27b8d9142-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.451447 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.451425    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24bc\" (UniqueName: \"kubernetes.io/projected/b04c2df7-04c9-416f-9da1-99d27b8d9142-kube-api-access-s24bc\") pod \"node-exporter-9cbmd\" (UID: \"b04c2df7-04c9-416f-9da1-99d27b8d9142\") " pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.479015 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.478849    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cbmd"
Sep 16 18:46:27.498023 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:27.497988    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04c2df7_04c9_416f_9da1_99d27b8d9142.slice/crio-90e285e61e2822a43854642c818ceefaea1312b3739159c0a58bb7d4f8829304 WatchSource:0}: Error finding container 90e285e61e2822a43854642c818ceefaea1312b3739159c0a58bb7d4f8829304: Status 404 returned error can't find the container with id 90e285e61e2822a43854642c818ceefaea1312b3739159c0a58bb7d4f8829304
Sep 16 18:46:27.693729 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:27.693516    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn"]
Sep 16 18:46:27.695468 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:27.695445    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5862522_1ff7_47d1_b29d_24a445296331.slice/crio-2fb9a714fece64ce093791befb487421e13ca24629c2cb9c70cbb382c40e76ba WatchSource:0}: Error finding container 2fb9a714fece64ce093791befb487421e13ca24629c2cb9c70cbb382c40e76ba: Status 404 returned error can't find the container with id 2fb9a714fece64ce093791befb487421e13ca24629c2cb9c70cbb382c40e76ba
Sep 16 18:46:28.349204 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:28.349168    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn" event={"ID":"f5862522-1ff7-47d1-b29d-24a445296331","Type":"ContainerStarted","Data":"e47c3f2e48db01661d0d87f93ee678e576a5df3d9479ef47c72acbb590e69bcb"}
Sep 16 18:46:28.349691 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:28.349297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn" event={"ID":"f5862522-1ff7-47d1-b29d-24a445296331","Type":"ContainerStarted","Data":"65d8900c3989926fe7e8d4f29e59fba66d8d3d311b1a71ce0eb6b7714c639520"}
Sep 16 18:46:28.349691 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:28.349338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn" event={"ID":"f5862522-1ff7-47d1-b29d-24a445296331","Type":"ContainerStarted","Data":"2fb9a714fece64ce093791befb487421e13ca24629c2cb9c70cbb382c40e76ba"}
Sep 16 18:46:28.350782 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:28.350757    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cbmd" event={"ID":"b04c2df7-04c9-416f-9da1-99d27b8d9142","Type":"ContainerStarted","Data":"90e285e61e2822a43854642c818ceefaea1312b3739159c0a58bb7d4f8829304"}
Sep 16 18:46:29.357862 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:29.357836    2584 generic.go:356] "Generic (PLEG): container finished" podID="b04c2df7-04c9-416f-9da1-99d27b8d9142" containerID="c8827a6f37d53f48eabfb8907fc8d8d0322f4e4e6e859956ca3f3606626ccbf7" exitCode=0
Sep 16 18:46:29.358219 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:29.357902    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cbmd" event={"ID":"b04c2df7-04c9-416f-9da1-99d27b8d9142","Type":"ContainerDied","Data":"c8827a6f37d53f48eabfb8907fc8d8d0322f4e4e6e859956ca3f3606626ccbf7"}
Sep 16 18:46:30.370446 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:30.370404    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn" event={"ID":"f5862522-1ff7-47d1-b29d-24a445296331","Type":"ContainerStarted","Data":"586f849180c893354a65675e440f02cf1e1c61ce2b13f267da482cd3d0f387ab"}
Sep 16 18:46:30.373490 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:30.373453    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cbmd" event={"ID":"b04c2df7-04c9-416f-9da1-99d27b8d9142","Type":"ContainerStarted","Data":"16c82f8c46ec6edfa6d6365ceb7c5cbd7fdc6cba6d105572d292ccefd7464bf8"}
Sep 16 18:46:30.373625 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:30.373504    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cbmd" event={"ID":"b04c2df7-04c9-416f-9da1-99d27b8d9142","Type":"ContainerStarted","Data":"000694561cb2466d5cab9c72ba39d28a2ee279635caf46b0be27fccef9689512"}
Sep 16 18:46:30.403779 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:30.403740    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-lj8dn" podStartSLOduration=1.9129114249999999 podStartE2EDuration="3.403727014s" totalImagesPullingTime="1.490815589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:30.403283979 +0000 UTC m=+121.070223125" watchObservedRunningTime="2026-09-16 18:46:30.403727014 +0000 UTC m=+121.070666159"
Sep 16 18:46:30.439729 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:30.439683    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9cbmd" podStartSLOduration=2.25383874 podStartE2EDuration="3.439668293s" totalImagesPullingTime="1.122887475s" totalInitContainerRuntime="62.942078ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:30.438465553 +0000 UTC m=+121.105404700" watchObservedRunningTime="2026-09-16 18:46:30.439668293 +0000 UTC m=+121.106607439"
Sep 16 18:46:37.299427 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:37.299396    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-549f4b54cd-lvknp"
Sep 16 18:46:39.635716 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:39.635676    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:46:39.637938 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:39.637916    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8f4d51-92a4-4bbe-9201-1ec009f2381b-metrics-certs\") pod \"network-metrics-daemon-lcxfx\" (UID: \"1f8f4d51-92a4-4bbe-9201-1ec009f2381b\") " pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:46:39.922225 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:39.922203    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rlx6q\""
Sep 16 18:46:39.930423 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:39.930401    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcxfx"
Sep 16 18:46:40.204115 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:40.204089    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lcxfx"]
Sep 16 18:46:40.206504 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:40.206476    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8f4d51_92a4_4bbe_9201_1ec009f2381b.slice/crio-e278ececa9bcd8d60d51a1183faacae7f44a0aaad84672ae803cb758279ec8b8 WatchSource:0}: Error finding container e278ececa9bcd8d60d51a1183faacae7f44a0aaad84672ae803cb758279ec8b8: Status 404 returned error can't find the container with id e278ececa9bcd8d60d51a1183faacae7f44a0aaad84672ae803cb758279ec8b8
Sep 16 18:46:40.410141 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:40.410096    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lcxfx" event={"ID":"1f8f4d51-92a4-4bbe-9201-1ec009f2381b","Type":"ContainerStarted","Data":"e278ececa9bcd8d60d51a1183faacae7f44a0aaad84672ae803cb758279ec8b8"}
Sep 16 18:46:41.085464 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.085432    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-lf2z7"]
Sep 16 18:46:41.089794 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.089773    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lf2z7"
Sep 16 18:46:41.092726 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.092709    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mc4l2\""
Sep 16 18:46:41.093118 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.093102    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:46:41.093409 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.093396    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:46:41.107377 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.107355    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lf2z7"]
Sep 16 18:46:41.249574 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.249540    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llfxv\" (UniqueName: \"kubernetes.io/projected/c34fffd4-724f-41cd-8277-f53e80ec0b57-kube-api-access-llfxv\") pod \"downloads-76f99584f4-lf2z7\" (UID: \"c34fffd4-724f-41cd-8277-f53e80ec0b57\") " pod="openshift-console/downloads-76f99584f4-lf2z7"
Sep 16 18:46:41.350560 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.350489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfxv\" (UniqueName: \"kubernetes.io/projected/c34fffd4-724f-41cd-8277-f53e80ec0b57-kube-api-access-llfxv\") pod \"downloads-76f99584f4-lf2z7\" (UID: \"c34fffd4-724f-41cd-8277-f53e80ec0b57\") " pod="openshift-console/downloads-76f99584f4-lf2z7"
Sep 16 18:46:41.360137 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.360113    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfxv\" (UniqueName: \"kubernetes.io/projected/c34fffd4-724f-41cd-8277-f53e80ec0b57-kube-api-access-llfxv\") pod \"downloads-76f99584f4-lf2z7\" (UID: \"c34fffd4-724f-41cd-8277-f53e80ec0b57\") " pod="openshift-console/downloads-76f99584f4-lf2z7"
Sep 16 18:46:41.399972 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.399945    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lf2z7"
Sep 16 18:46:41.416390 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.416368    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lcxfx" event={"ID":"1f8f4d51-92a4-4bbe-9201-1ec009f2381b","Type":"ContainerStarted","Data":"80ea6a70ff13aea2971f834a05dc631e8b3b1810c17ba199dfc65fdac722fc38"}
Sep 16 18:46:41.416493 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.416413    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lcxfx" event={"ID":"1f8f4d51-92a4-4bbe-9201-1ec009f2381b","Type":"ContainerStarted","Data":"b767f08656988d68630f7de918956553acc21fd999b979992af4a1b4f4d55066"}
Sep 16 18:46:41.439411 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.439353    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lcxfx" podStartSLOduration=130.525927777 podStartE2EDuration="2m11.439332923s" totalImagesPullingTime="913.405146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:41.438347996 +0000 UTC m=+132.105287171" watchObservedRunningTime="2026-09-16 18:46:41.439332923 +0000 UTC m=+132.106272071"
Sep 16 18:46:41.674848 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:41.674759    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lf2z7"]
Sep 16 18:46:41.676758 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:41.676733    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34fffd4_724f_41cd_8277_f53e80ec0b57.slice/crio-ea6afc8cb97ef69c7a026a2ed86084f7553a3d1524b11972c20d5d9fc2749520 WatchSource:0}: Error finding container ea6afc8cb97ef69c7a026a2ed86084f7553a3d1524b11972c20d5d9fc2749520: Status 404 returned error can't find the container with id ea6afc8cb97ef69c7a026a2ed86084f7553a3d1524b11972c20d5d9fc2749520
Sep 16 18:46:42.422864 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:42.422829    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lf2z7" event={"ID":"c34fffd4-724f-41cd-8277-f53e80ec0b57","Type":"ContainerStarted","Data":"ea6afc8cb97ef69c7a026a2ed86084f7553a3d1524b11972c20d5d9fc2749520"}
Sep 16 18:46:46.551547 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.551507    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8b5f7d56b-g4jh9"]
Sep 16 18:46:46.555010 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.554987    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.557703 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.557680    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:46:46.558540 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.558518    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:46:46.558673 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.558579    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:46:46.558673 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.558621    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fx24k\""
Sep 16 18:46:46.558784 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.558690    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:46:46.558854 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.558837    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:46:46.567552 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.567529    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b5f7d56b-g4jh9"]
Sep 16 18:46:46.592394 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.592356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-config\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.592394 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.592389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss5x7\" (UniqueName: \"kubernetes.io/projected/2b2debcc-7821-4b01-9371-72bbde1c23c0-kube-api-access-ss5x7\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.592549 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.592440    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-serving-cert\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.592549 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.592476    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-oauth-config\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.592549 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.592501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-service-ca\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.592662 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.592552    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-oauth-serving-cert\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.693194 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.693155    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-serving-cert\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.693383 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.693202    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-oauth-config\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.693383 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.693237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-service-ca\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.693383 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.693292    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-oauth-serving-cert\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.693383 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.693333    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-config\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.693383 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.693358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss5x7\" (UniqueName: \"kubernetes.io/projected/2b2debcc-7821-4b01-9371-72bbde1c23c0-kube-api-access-ss5x7\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.694134 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.694098    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-service-ca\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.694288 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.694146    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-oauth-serving-cert\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.694288 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.694172    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-config\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.696107 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.696083    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-oauth-config\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.696202 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.696093    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-serving-cert\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.702079 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.702055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss5x7\" (UniqueName: \"kubernetes.io/projected/2b2debcc-7821-4b01-9371-72bbde1c23c0-kube-api-access-ss5x7\") pod \"console-8b5f7d56b-g4jh9\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") " pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:46.866890 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:46.866808    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:47.160662 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:47.160441    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b5f7d56b-g4jh9"]
Sep 16 18:46:47.163410 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:47.163381    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2debcc_7821_4b01_9371_72bbde1c23c0.slice/crio-cdb58a8aa4f186017d2bb8351472010979c84209c1baf22432a2194036f11425 WatchSource:0}: Error finding container cdb58a8aa4f186017d2bb8351472010979c84209c1baf22432a2194036f11425: Status 404 returned error can't find the container with id cdb58a8aa4f186017d2bb8351472010979c84209c1baf22432a2194036f11425
Sep 16 18:46:47.447368 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:47.447299    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b5f7d56b-g4jh9" event={"ID":"2b2debcc-7821-4b01-9371-72bbde1c23c0","Type":"ContainerStarted","Data":"cdb58a8aa4f186017d2bb8351472010979c84209c1baf22432a2194036f11425"}
Sep 16 18:46:50.464179 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:50.464118    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b5f7d56b-g4jh9" event={"ID":"2b2debcc-7821-4b01-9371-72bbde1c23c0","Type":"ContainerStarted","Data":"da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992"}
Sep 16 18:46:50.486465 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:50.486374    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8b5f7d56b-g4jh9" podStartSLOduration=1.415702925 podStartE2EDuration="4.486353902s" totalImagesPullingTime="3.070650977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:50.483381663 +0000 UTC m=+141.150320876" watchObservedRunningTime="2026-09-16 18:46:50.486353902 +0000 UTC m=+141.153293049"
Sep 16 18:46:56.188707 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.188673    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cc475db8d-svtp6"]
Sep 16 18:46:56.192209 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.192189    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.201008 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.200985    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:46:56.204095 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.204073    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc475db8d-svtp6"]
Sep 16 18:46:56.272208 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.272172    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-trusted-ca-bundle\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.272390 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.272217    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-oauth-config\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.272390 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.272254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-config\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.272390 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.272347    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-serving-cert\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.272390 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.272375    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-oauth-serving-cert\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.272613 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.272404    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-service-ca\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.272613 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.272473    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbpns\" (UniqueName: \"kubernetes.io/projected/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-kube-api-access-lbpns\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.373602 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.373556    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-trusted-ca-bundle\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.373602 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.373609    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-oauth-config\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.373844 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.373654    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-config\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.373844 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.373709    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-serving-cert\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.373844 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.373730    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-oauth-serving-cert\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.373844 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.373760    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-service-ca\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.373844 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.373788    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpns\" (UniqueName: \"kubernetes.io/projected/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-kube-api-access-lbpns\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.374569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.374515    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-config\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.374569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.374535    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-trusted-ca-bundle\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.374759 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.374739    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-oauth-serving-cert\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.374875 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.374843    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-service-ca\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.377002 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.376977    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-oauth-config\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.377159 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.377138    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-serving-cert\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.382459 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.382437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpns\" (UniqueName: \"kubernetes.io/projected/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-kube-api-access-lbpns\") pod \"console-7cc475db8d-svtp6\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") " pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.502529 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.502453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:46:56.867463 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.867374    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:56.867717 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.867693    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:56.873106 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:56.873083    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:57.498754 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:57.498724    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:46:57.885812 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:57.885788    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc475db8d-svtp6"]
Sep 16 18:46:57.888360 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:46:57.888326    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf9f2ee_638b_4df3_ae5f_0e4d3cf4c157.slice/crio-14315050a999466b8242cf5bead2dd9567fc1e838ced7e95bf6a679fa11913a2 WatchSource:0}: Error finding container 14315050a999466b8242cf5bead2dd9567fc1e838ced7e95bf6a679fa11913a2: Status 404 returned error can't find the container with id 14315050a999466b8242cf5bead2dd9567fc1e838ced7e95bf6a679fa11913a2
Sep 16 18:46:58.500120 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.500082    2584 generic.go:356] "Generic (PLEG): container finished" podID="3674164a-4141-4d00-ba6a-45fff63302fb" containerID="efbcc8aab96890765f219d038e05be74e82461ebd54821e603a2b2fb07702f00" exitCode=0
Sep 16 18:46:58.500622 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.500163    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h97rl" event={"ID":"3674164a-4141-4d00-ba6a-45fff63302fb","Type":"ContainerDied","Data":"efbcc8aab96890765f219d038e05be74e82461ebd54821e603a2b2fb07702f00"}
Sep 16 18:46:58.500622 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.500606    2584 scope.go:117] "RemoveContainer" containerID="efbcc8aab96890765f219d038e05be74e82461ebd54821e603a2b2fb07702f00"
Sep 16 18:46:58.502144 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.502031    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lf2z7" event={"ID":"c34fffd4-724f-41cd-8277-f53e80ec0b57","Type":"ContainerStarted","Data":"0f0780f9fff0c86fa1dd020cca4b9e1840bbd51f1d263028a92bc87546ec2a04"}
Sep 16 18:46:58.502343 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.502321    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-lf2z7"
Sep 16 18:46:58.504930 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.504484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc475db8d-svtp6" event={"ID":"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157","Type":"ContainerStarted","Data":"60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365"}
Sep 16 18:46:58.504930 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.504530    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc475db8d-svtp6" event={"ID":"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157","Type":"ContainerStarted","Data":"14315050a999466b8242cf5bead2dd9567fc1e838ced7e95bf6a679fa11913a2"}
Sep 16 18:46:58.513549 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.513529    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-lf2z7"
Sep 16 18:46:58.540957 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.540905    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cc475db8d-svtp6" podStartSLOduration=2.540884961 podStartE2EDuration="2.540884961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:58.539645233 +0000 UTC m=+149.206584402" watchObservedRunningTime="2026-09-16 18:46:58.540884961 +0000 UTC m=+149.207824103"
Sep 16 18:46:58.557688 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:58.557639    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-lf2z7" podStartSLOduration=1.526198325 podStartE2EDuration="17.557618146s" totalImagesPullingTime="16.031419821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:58.557460299 +0000 UTC m=+149.224399474" watchObservedRunningTime="2026-09-16 18:46:58.557618146 +0000 UTC m=+149.224557293"
Sep 16 18:46:59.511816 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:46:59.511773    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h97rl" event={"ID":"3674164a-4141-4d00-ba6a-45fff63302fb","Type":"ContainerStarted","Data":"abfd271061c4dfc492c17077dd0fb16f2bf1dfee9542cfa438c9383c072a5257"}
Sep 16 18:47:02.444095 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:02.444063    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cbmd_b04c2df7-04c9-416f-9da1-99d27b8d9142/init-textfile/0.log"
Sep 16 18:47:02.648794 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:02.648765    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cbmd_b04c2df7-04c9-416f-9da1-99d27b8d9142/node-exporter/0.log"
Sep 16 18:47:02.845579 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:02.845497    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cbmd_b04c2df7-04c9-416f-9da1-99d27b8d9142/kube-rbac-proxy/0.log"
Sep 16 18:47:03.644915 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:03.644851    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lj8dn_f5862522-1ff7-47d1-b29d-24a445296331/kube-rbac-proxy-main/0.log"
Sep 16 18:47:03.844701 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:03.844676    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lj8dn_f5862522-1ff7-47d1-b29d-24a445296331/kube-rbac-proxy-self/0.log"
Sep 16 18:47:04.047642 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:04.047611    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lj8dn_f5862522-1ff7-47d1-b29d-24a445296331/openshift-state-metrics/0.log"
Sep 16 18:47:05.645480 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:05.645447    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gn8d8_7d4bf852-bbc0-4429-a0e6-c9a723915f11/prometheus-operator/0.log"
Sep 16 18:47:05.844095 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:05.844069    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gn8d8_7d4bf852-bbc0-4429-a0e6-c9a723915f11/kube-rbac-proxy/0.log"
Sep 16 18:47:06.502978 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:06.502941    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:47:06.503229 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:06.503033    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:47:06.508187 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:06.508161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:47:06.546192 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:06.546170    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:47:06.609021 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:06.608993    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b5f7d56b-g4jh9"]
Sep 16 18:47:07.445499 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:07.445467    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-2275q_e13f7d12-6a82-469f-8c78-52ef0e7af2b3/networking-console-plugin/0.log"
Sep 16 18:47:08.045322 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:08.045293    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cc475db8d-svtp6_ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157/console/0.log"
Sep 16 18:47:08.244796 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:08.244768    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8b5f7d56b-g4jh9_2b2debcc-7821-4b01-9371-72bbde1c23c0/console/0.log"
Sep 16 18:47:08.444851 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:08.444824    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-lf2z7_c34fffd4-724f-41cd-8277-f53e80ec0b57/download-server/0.log"
Sep 16 18:47:17.596226 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:17.596191    2584 generic.go:356] "Generic (PLEG): container finished" podID="6678e175-f02f-43f4-9a15-53efd70313a1" containerID="630b0ef3950363e8a501f5e485ce558187e1e4fae7ffcbcd5ad2aba71c79afe0" exitCode=0
Sep 16 18:47:17.596660 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:17.596236    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h" event={"ID":"6678e175-f02f-43f4-9a15-53efd70313a1","Type":"ContainerDied","Data":"630b0ef3950363e8a501f5e485ce558187e1e4fae7ffcbcd5ad2aba71c79afe0"}
Sep 16 18:47:17.596726 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:17.596675    2584 scope.go:117] "RemoveContainer" containerID="630b0ef3950363e8a501f5e485ce558187e1e4fae7ffcbcd5ad2aba71c79afe0"
Sep 16 18:47:18.602355 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:18.602324    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xmn8h" event={"ID":"6678e175-f02f-43f4-9a15-53efd70313a1","Type":"ContainerStarted","Data":"37ba7cb029418becb701025bac81c62c193abb04ad6c0b7652df802a280f4084"}
Sep 16 18:47:31.632916 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:31.632808    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8b5f7d56b-g4jh9" podUID="2b2debcc-7821-4b01-9371-72bbde1c23c0" containerName="console" containerID="cri-o://da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992" gracePeriod=15
Sep 16 18:47:31.913039 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:31.913012    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:47:32.078780 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.078752    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-config\") pod \"2b2debcc-7821-4b01-9371-72bbde1c23c0\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") "
Sep 16 18:47:32.078963 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.078787    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-oauth-config\") pod \"2b2debcc-7821-4b01-9371-72bbde1c23c0\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") "
Sep 16 18:47:32.078963 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.078829    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-serving-cert\") pod \"2b2debcc-7821-4b01-9371-72bbde1c23c0\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") "
Sep 16 18:47:32.078963 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.078952    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-oauth-serving-cert\") pod \"2b2debcc-7821-4b01-9371-72bbde1c23c0\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") "
Sep 16 18:47:32.079123 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.079025    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss5x7\" (UniqueName: \"kubernetes.io/projected/2b2debcc-7821-4b01-9371-72bbde1c23c0-kube-api-access-ss5x7\") pod \"2b2debcc-7821-4b01-9371-72bbde1c23c0\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") "
Sep 16 18:47:32.079123 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.079047    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-service-ca\") pod \"2b2debcc-7821-4b01-9371-72bbde1c23c0\" (UID: \"2b2debcc-7821-4b01-9371-72bbde1c23c0\") "
Sep 16 18:47:32.079228 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.079181    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-config" (OuterVolumeSpecName: "console-config") pod "2b2debcc-7821-4b01-9371-72bbde1c23c0" (UID: "2b2debcc-7821-4b01-9371-72bbde1c23c0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:32.079320 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.079299    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:47:32.079377 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.079304    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b2debcc-7821-4b01-9371-72bbde1c23c0" (UID: "2b2debcc-7821-4b01-9371-72bbde1c23c0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:32.079525 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.079495    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b2debcc-7821-4b01-9371-72bbde1c23c0" (UID: "2b2debcc-7821-4b01-9371-72bbde1c23c0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:32.081070 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.081044    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b2debcc-7821-4b01-9371-72bbde1c23c0" (UID: "2b2debcc-7821-4b01-9371-72bbde1c23c0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:32.081153 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.081094    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b2debcc-7821-4b01-9371-72bbde1c23c0" (UID: "2b2debcc-7821-4b01-9371-72bbde1c23c0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:32.081153 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.081105    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2debcc-7821-4b01-9371-72bbde1c23c0-kube-api-access-ss5x7" (OuterVolumeSpecName: "kube-api-access-ss5x7") pod "2b2debcc-7821-4b01-9371-72bbde1c23c0" (UID: "2b2debcc-7821-4b01-9371-72bbde1c23c0"). InnerVolumeSpecName "kube-api-access-ss5x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:32.180320 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.180296    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-oauth-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:47:32.180320 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.180316    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b2debcc-7821-4b01-9371-72bbde1c23c0-console-serving-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:47:32.180439 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.180326    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-oauth-serving-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:47:32.180439 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.180335    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss5x7\" (UniqueName: \"kubernetes.io/projected/2b2debcc-7821-4b01-9371-72bbde1c23c0-kube-api-access-ss5x7\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:47:32.180439 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.180345    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2debcc-7821-4b01-9371-72bbde1c23c0-service-ca\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:47:32.654559 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.654481    2584 generic.go:356] "Generic (PLEG): container finished" podID="2b2debcc-7821-4b01-9371-72bbde1c23c0" containerID="da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992" exitCode=0
Sep 16 18:47:32.654559 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.654525    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b5f7d56b-g4jh9" event={"ID":"2b2debcc-7821-4b01-9371-72bbde1c23c0","Type":"ContainerDied","Data":"da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992"}
Sep 16 18:47:32.654559 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.654541    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b5f7d56b-g4jh9"
Sep 16 18:47:32.654960 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.654605    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b5f7d56b-g4jh9" event={"ID":"2b2debcc-7821-4b01-9371-72bbde1c23c0","Type":"ContainerDied","Data":"cdb58a8aa4f186017d2bb8351472010979c84209c1baf22432a2194036f11425"}
Sep 16 18:47:32.664503 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.664479    2584 scope.go:117] "RemoveContainer" containerID="da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992"
Sep 16 18:47:32.682558 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.682542    2584 scope.go:117] "RemoveContainer" containerID="da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992"
Sep 16 18:47:32.682834 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:47:32.682812    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992\": container with ID starting with da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992 not found: ID does not exist" containerID="da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992"
Sep 16 18:47:32.682888 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.682841    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992"} err="failed to get container status \"da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992\": rpc error: code = NotFound desc = could not find container \"da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992\": container with ID starting with da7e0f5ededd67277c0a1d92db3c0ccfebecdb40f2fbf945c70692f4d84d4992 not found: ID does not exist"
Sep 16 18:47:32.687165 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.687146    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8b5f7d56b-g4jh9"]
Sep 16 18:47:32.691637 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:32.691614    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8b5f7d56b-g4jh9"]
Sep 16 18:47:33.815439 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:33.815405    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2debcc-7821-4b01-9371-72bbde1c23c0" path="/var/lib/kubelet/pods/2b2debcc-7821-4b01-9371-72bbde1c23c0/volumes"
Sep 16 18:47:55.526399 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.526367    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d8b9d7cbc-knvxp"]
Sep 16 18:47:55.526861 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.526669    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2debcc-7821-4b01-9371-72bbde1c23c0" containerName="console"
Sep 16 18:47:55.526861 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.526681    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2debcc-7821-4b01-9371-72bbde1c23c0" containerName="console"
Sep 16 18:47:55.526861 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.526736    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2debcc-7821-4b01-9371-72bbde1c23c0" containerName="console"
Sep 16 18:47:55.529490 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.529473    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.537450 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.537427    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-trusted-ca-bundle\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.537559 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.537466    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-config\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.537559 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.537516    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-oauth-config\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.537559 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.537533    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-service-ca\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.537559 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.537553    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskp5\" (UniqueName: \"kubernetes.io/projected/448a802b-3b71-4347-ba1c-4bc90364f3a2-kube-api-access-bskp5\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.537696 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.537628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-oauth-serving-cert\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.537696 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.537661    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-serving-cert\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.638284 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.638234    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-oauth-serving-cert\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.638284 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.638290    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-serving-cert\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.638506 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.638314    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-trusted-ca-bundle\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.638506 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.638339    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-config\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.638506 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.638369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-oauth-config\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.638506 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.638386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-service-ca\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.638506 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.638403    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskp5\" (UniqueName: \"kubernetes.io/projected/448a802b-3b71-4347-ba1c-4bc90364f3a2-kube-api-access-bskp5\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.639143 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.639112    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-oauth-serving-cert\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.639251 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.639112    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-config\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.639326 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.639251    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-trusted-ca-bundle\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.639364 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.639338    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-service-ca\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.640782 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.640761    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-serving-cert\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.640867 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.640791    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-oauth-config\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.654000 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.653978    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskp5\" (UniqueName: \"kubernetes.io/projected/448a802b-3b71-4347-ba1c-4bc90364f3a2-kube-api-access-bskp5\") pod \"console-5d8b9d7cbc-knvxp\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") " pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:55.687814 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.687786    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8b9d7cbc-knvxp"]
Sep 16 18:47:55.838215 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:55.838133    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:47:56.132431 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:56.132352    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8b9d7cbc-knvxp"]
Sep 16 18:47:56.136497 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:47:56.136472    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448a802b_3b71_4347_ba1c_4bc90364f3a2.slice/crio-7220ba94f163c23a4696d227e37f912389645d8a45265154fd28a73a0f1c9830 WatchSource:0}: Error finding container 7220ba94f163c23a4696d227e37f912389645d8a45265154fd28a73a0f1c9830: Status 404 returned error can't find the container with id 7220ba94f163c23a4696d227e37f912389645d8a45265154fd28a73a0f1c9830
Sep 16 18:47:56.742846 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:56.742810    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b9d7cbc-knvxp" event={"ID":"448a802b-3b71-4347-ba1c-4bc90364f3a2","Type":"ContainerStarted","Data":"a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2"}
Sep 16 18:47:56.743242 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:56.742856    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b9d7cbc-knvxp" event={"ID":"448a802b-3b71-4347-ba1c-4bc90364f3a2","Type":"ContainerStarted","Data":"7220ba94f163c23a4696d227e37f912389645d8a45265154fd28a73a0f1c9830"}
Sep 16 18:47:56.784394 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:47:56.784344    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d8b9d7cbc-knvxp" podStartSLOduration=1.784331248 podStartE2EDuration="1.784331248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:56.783730147 +0000 UTC m=+207.450669312" watchObservedRunningTime="2026-09-16 18:47:56.784331248 +0000 UTC m=+207.451270394"
Sep 16 18:48:05.838849 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:05.838813    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:48:05.839447 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:05.838877    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:48:05.843501 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:05.843478    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:48:06.783924 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:06.783895    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:48:06.849740 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:06.849713    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc475db8d-svtp6"]
Sep 16 18:48:31.870680 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:31.870639    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cc475db8d-svtp6" podUID="ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" containerName="console" containerID="cri-o://60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365" gracePeriod=15
Sep 16 18:48:32.112711 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.112688    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:48:32.204295 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.204254    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-serving-cert\") pod \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") "
Sep 16 18:48:32.204458 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.204325    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-oauth-config\") pod \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") "
Sep 16 18:48:32.204458 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.204414    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-config\") pod \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") "
Sep 16 18:48:32.204458 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.204443    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbpns\" (UniqueName: \"kubernetes.io/projected/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-kube-api-access-lbpns\") pod \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") "
Sep 16 18:48:32.204625 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.204477    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-oauth-serving-cert\") pod \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") "
Sep 16 18:48:32.204625 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.204521    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-service-ca\") pod \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") "
Sep 16 18:48:32.204625 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.204540    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-trusted-ca-bundle\") pod \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\" (UID: \"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157\") "
Sep 16 18:48:32.204818 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.204792    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-config" (OuterVolumeSpecName: "console-config") pod "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" (UID: "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:32.204954 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.204866    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-service-ca" (OuterVolumeSpecName: "service-ca") pod "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" (UID: "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:32.205043 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.205018    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" (UID: "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:32.205100 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.205066    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" (UID: "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:32.206475 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.206450    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-kube-api-access-lbpns" (OuterVolumeSpecName: "kube-api-access-lbpns") pod "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" (UID: "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157"). InnerVolumeSpecName "kube-api-access-lbpns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:32.206563 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.206473    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" (UID: "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:32.206563 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.206522    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" (UID: "ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:32.305888 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.305859    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-serving-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.306015 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.305903    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-oauth-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.306015 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.305918    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-console-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.306015 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.305932    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbpns\" (UniqueName: \"kubernetes.io/projected/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-kube-api-access-lbpns\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.306015 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.305946    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-oauth-serving-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.310171 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.310139    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-service-ca\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.310171 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.310168    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157-trusted-ca-bundle\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.881768 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.881728    2584 generic.go:356] "Generic (PLEG): container finished" podID="ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" containerID="60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365" exitCode=0
Sep 16 18:48:32.882204 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.881790    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc475db8d-svtp6"
Sep 16 18:48:32.882204 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.881815    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc475db8d-svtp6" event={"ID":"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157","Type":"ContainerDied","Data":"60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365"}
Sep 16 18:48:32.882204 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.881886    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc475db8d-svtp6" event={"ID":"ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157","Type":"ContainerDied","Data":"14315050a999466b8242cf5bead2dd9567fc1e838ced7e95bf6a679fa11913a2"}
Sep 16 18:48:32.882204 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.881918    2584 scope.go:117] "RemoveContainer" containerID="60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365"
Sep 16 18:48:32.891012 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.890994    2584 scope.go:117] "RemoveContainer" containerID="60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365"
Sep 16 18:48:32.891282 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:48:32.891245    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365\": container with ID starting with 60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365 not found: ID does not exist" containerID="60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365"
Sep 16 18:48:32.891337 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.891281    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365"} err="failed to get container status \"60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365\": rpc error: code = NotFound desc = could not find container \"60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365\": container with ID starting with 60304b4644d2e2dfa6fbc7ad6093bcbaa29d08fb02451031f06935f96038f365 not found: ID does not exist"
Sep 16 18:48:32.905068 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.905046    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc475db8d-svtp6"]
Sep 16 18:48:32.914325 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:32.914304    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cc475db8d-svtp6"]
Sep 16 18:48:33.815899 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:48:33.815868    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" path="/var/lib/kubelet/pods/ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157/volumes"
Sep 16 18:49:29.761721 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:29.761691    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:49:41.099233 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.099200    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59b9bf4847-j9822"]
Sep 16 18:49:41.101533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.099508    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" containerName="console"
Sep 16 18:49:41.101533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.099519    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" containerName="console"
Sep 16 18:49:41.101533 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.099565    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf9f2ee-638b-4df3-ae5f-0e4d3cf4c157" containerName="console"
Sep 16 18:49:41.102415 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.102397    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.114961 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.114938    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b9bf4847-j9822"]
Sep 16 18:49:41.200880 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.200848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-service-ca\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.200880 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.200880    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-oauth-serving-cert\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.201120 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.200917    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-config\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.201120 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.200984    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-oauth-config\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.201120 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.201022    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-serving-cert\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.201120 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.201038    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-trusted-ca-bundle\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.201120 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.201063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86zz\" (UniqueName: \"kubernetes.io/projected/5d335e37-5496-4c3d-bc25-a292e7b5865a-kube-api-access-d86zz\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.301515 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.301480    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-config\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.301672 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.301519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-oauth-config\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.301672 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.301549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-serving-cert\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.301672 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.301571    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-trusted-ca-bundle\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.301672 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.301607    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86zz\" (UniqueName: \"kubernetes.io/projected/5d335e37-5496-4c3d-bc25-a292e7b5865a-kube-api-access-d86zz\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.301672 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.301643    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-service-ca\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.301672 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.301663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-oauth-serving-cert\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.302386 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.302359    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-config\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.302556 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.302534    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-oauth-serving-cert\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.302604 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.302561    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-service-ca\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.302604 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.302570    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-trusted-ca-bundle\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.304022 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.304000    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-serving-cert\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.304170 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.304147    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-oauth-config\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.311466 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.311446    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86zz\" (UniqueName: \"kubernetes.io/projected/5d335e37-5496-4c3d-bc25-a292e7b5865a-kube-api-access-d86zz\") pod \"console-59b9bf4847-j9822\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") " pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.411543 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.411521    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:41.684299 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:41.684186    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b9bf4847-j9822"]
Sep 16 18:49:41.686849 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:49:41.686816    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d335e37_5496_4c3d_bc25_a292e7b5865a.slice/crio-8daa14cc967c028a0ab01eacc2cb9e7f17254b2818c1acfc3e1cc521e1e488e5 WatchSource:0}: Error finding container 8daa14cc967c028a0ab01eacc2cb9e7f17254b2818c1acfc3e1cc521e1e488e5: Status 404 returned error can't find the container with id 8daa14cc967c028a0ab01eacc2cb9e7f17254b2818c1acfc3e1cc521e1e488e5
Sep 16 18:49:42.134621 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:42.134536    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b9bf4847-j9822" event={"ID":"5d335e37-5496-4c3d-bc25-a292e7b5865a","Type":"ContainerStarted","Data":"63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b"}
Sep 16 18:49:42.134621 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:42.134597    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b9bf4847-j9822" event={"ID":"5d335e37-5496-4c3d-bc25-a292e7b5865a","Type":"ContainerStarted","Data":"8daa14cc967c028a0ab01eacc2cb9e7f17254b2818c1acfc3e1cc521e1e488e5"}
Sep 16 18:49:42.154162 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:42.154117    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59b9bf4847-j9822" podStartSLOduration=1.154103666 podStartE2EDuration="1.154103666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:42.152404315 +0000 UTC m=+312.819343463" watchObservedRunningTime="2026-09-16 18:49:42.154103666 +0000 UTC m=+312.821042813"
Sep 16 18:49:51.411937 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:51.411897    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:51.412389 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:51.411954    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:51.416575 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:51.416556    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:52.176610 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:52.176584    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 18:49:52.226049 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:49:52.226016    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8b9d7cbc-knvxp"]
Sep 16 18:50:13.399873 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.399839    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"]
Sep 16 18:50:13.403296 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.403278    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.406019 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.405997    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:50:13.406249 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.406232    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lr77n\""
Sep 16 18:50:13.407112 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.407094    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:50:13.413404 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.413385    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"]
Sep 16 18:50:13.556630 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.556595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7rp\" (UniqueName: \"kubernetes.io/projected/c6646064-e1b1-4aef-ba29-bc599e34a0fa-kube-api-access-lm7rp\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.556745 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.556647    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.556745 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.556669    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.657923 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.657861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.657923 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.657921    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7rp\" (UniqueName: \"kubernetes.io/projected/c6646064-e1b1-4aef-ba29-bc599e34a0fa-kube-api-access-lm7rp\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.658081 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.657955    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.658204 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.658184    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.658293 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.658219    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.667668 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.667645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7rp\" (UniqueName: \"kubernetes.io/projected/c6646064-e1b1-4aef-ba29-bc599e34a0fa-kube-api-access-lm7rp\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.712683 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.712662    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:13.982491 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.982468    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"]
Sep 16 18:50:13.984459 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:50:13.984431    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6646064_e1b1_4aef_ba29_bc599e34a0fa.slice/crio-9b7f14d13fe5f7485da304325d08c8c05735b3571901d028477af4aa0538bc1d WatchSource:0}: Error finding container 9b7f14d13fe5f7485da304325d08c8c05735b3571901d028477af4aa0538bc1d: Status 404 returned error can't find the container with id 9b7f14d13fe5f7485da304325d08c8c05735b3571901d028477af4aa0538bc1d
Sep 16 18:50:13.986360 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:13.986345    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:14.256647 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:14.256577    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd" event={"ID":"c6646064-e1b1-4aef-ba29-bc599e34a0fa","Type":"ContainerStarted","Data":"9b7f14d13fe5f7485da304325d08c8c05735b3571901d028477af4aa0538bc1d"}
Sep 16 18:50:17.246359 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.246322    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d8b9d7cbc-knvxp" podUID="448a802b-3b71-4347-ba1c-4bc90364f3a2" containerName="console" containerID="cri-o://a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2" gracePeriod=15
Sep 16 18:50:17.490095 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.490066    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:50:17.594288 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594168    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-trusted-ca-bundle\") pod \"448a802b-3b71-4347-ba1c-4bc90364f3a2\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") "
Sep 16 18:50:17.594288 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594229    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-service-ca\") pod \"448a802b-3b71-4347-ba1c-4bc90364f3a2\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") "
Sep 16 18:50:17.594288 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594261    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bskp5\" (UniqueName: \"kubernetes.io/projected/448a802b-3b71-4347-ba1c-4bc90364f3a2-kube-api-access-bskp5\") pod \"448a802b-3b71-4347-ba1c-4bc90364f3a2\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") "
Sep 16 18:50:17.594696 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594321    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-serving-cert\") pod \"448a802b-3b71-4347-ba1c-4bc90364f3a2\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") "
Sep 16 18:50:17.594696 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594369    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-oauth-config\") pod \"448a802b-3b71-4347-ba1c-4bc90364f3a2\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") "
Sep 16 18:50:17.594696 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594405    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-config\") pod \"448a802b-3b71-4347-ba1c-4bc90364f3a2\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") "
Sep 16 18:50:17.594696 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594443    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-oauth-serving-cert\") pod \"448a802b-3b71-4347-ba1c-4bc90364f3a2\" (UID: \"448a802b-3b71-4347-ba1c-4bc90364f3a2\") "
Sep 16 18:50:17.594696 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594652    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "448a802b-3b71-4347-ba1c-4bc90364f3a2" (UID: "448a802b-3b71-4347-ba1c-4bc90364f3a2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:17.594696 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594679    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-service-ca" (OuterVolumeSpecName: "service-ca") pod "448a802b-3b71-4347-ba1c-4bc90364f3a2" (UID: "448a802b-3b71-4347-ba1c-4bc90364f3a2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:17.595104 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594932    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "448a802b-3b71-4347-ba1c-4bc90364f3a2" (UID: "448a802b-3b71-4347-ba1c-4bc90364f3a2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:17.595104 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.594959    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-config" (OuterVolumeSpecName: "console-config") pod "448a802b-3b71-4347-ba1c-4bc90364f3a2" (UID: "448a802b-3b71-4347-ba1c-4bc90364f3a2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:17.597079 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.597036    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "448a802b-3b71-4347-ba1c-4bc90364f3a2" (UID: "448a802b-3b71-4347-ba1c-4bc90364f3a2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:17.597193 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.597082    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448a802b-3b71-4347-ba1c-4bc90364f3a2-kube-api-access-bskp5" (OuterVolumeSpecName: "kube-api-access-bskp5") pod "448a802b-3b71-4347-ba1c-4bc90364f3a2" (UID: "448a802b-3b71-4347-ba1c-4bc90364f3a2"). InnerVolumeSpecName "kube-api-access-bskp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:17.597261 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.597204    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "448a802b-3b71-4347-ba1c-4bc90364f3a2" (UID: "448a802b-3b71-4347-ba1c-4bc90364f3a2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:17.695838 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.695805    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-trusted-ca-bundle\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:17.695838 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.695832    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-service-ca\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:17.695838 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.695842    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bskp5\" (UniqueName: \"kubernetes.io/projected/448a802b-3b71-4347-ba1c-4bc90364f3a2-kube-api-access-bskp5\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:17.696061 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.695851    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-serving-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:17.696061 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.695860    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-oauth-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:17.696061 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.695868    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-console-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:17.696061 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:17.695876    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/448a802b-3b71-4347-ba1c-4bc90364f3a2-oauth-serving-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:18.275316 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:18.275250    2584 generic.go:356] "Generic (PLEG): container finished" podID="448a802b-3b71-4347-ba1c-4bc90364f3a2" containerID="a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2" exitCode=0
Sep 16 18:50:18.275316 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:18.275317    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b9d7cbc-knvxp" event={"ID":"448a802b-3b71-4347-ba1c-4bc90364f3a2","Type":"ContainerDied","Data":"a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2"}
Sep 16 18:50:18.275818 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:18.275335    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b9d7cbc-knvxp"
Sep 16 18:50:18.275818 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:18.275371    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b9d7cbc-knvxp" event={"ID":"448a802b-3b71-4347-ba1c-4bc90364f3a2","Type":"ContainerDied","Data":"7220ba94f163c23a4696d227e37f912389645d8a45265154fd28a73a0f1c9830"}
Sep 16 18:50:18.275818 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:18.275398    2584 scope.go:117] "RemoveContainer" containerID="a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2"
Sep 16 18:50:18.292446 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:18.292418    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8b9d7cbc-knvxp"]
Sep 16 18:50:18.295930 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:18.295906    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d8b9d7cbc-knvxp"]
Sep 16 18:50:19.629240 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:19.629064    2584 scope.go:117] "RemoveContainer" containerID="a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2"
Sep 16 18:50:19.629485 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:19.629439    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2\": container with ID starting with a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2 not found: ID does not exist" containerID="a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2"
Sep 16 18:50:19.629485 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:19.629467    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2"} err="failed to get container status \"a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2\": rpc error: code = NotFound desc = could not find container \"a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2\": container with ID starting with a2cc83d3230de1be65173aefc805dd5f50e492e9d63b9a69daf3929f5dbaa9e2 not found: ID does not exist"
Sep 16 18:50:19.815880 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:19.815854    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="448a802b-3b71-4347-ba1c-4bc90364f3a2" path="/var/lib/kubelet/pods/448a802b-3b71-4347-ba1c-4bc90364f3a2/volumes"
Sep 16 18:50:20.284970 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:20.284940    2584 generic.go:356] "Generic (PLEG): container finished" podID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerID="0d5dc6c10d97d274a63c2254618ceaa7dca8c89f532a31fe37bfdf167675a44a" exitCode=0
Sep 16 18:50:20.285103 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:20.285030    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd" event={"ID":"c6646064-e1b1-4aef-ba29-bc599e34a0fa","Type":"ContainerDied","Data":"0d5dc6c10d97d274a63c2254618ceaa7dca8c89f532a31fe37bfdf167675a44a"}
Sep 16 18:50:24.302821 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:24.302791    2584 generic.go:356] "Generic (PLEG): container finished" podID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerID="f7cd5504008523d802d056d65dec3e0df9ae90b6bf23a51a57e738adad7ce5d8" exitCode=0
Sep 16 18:50:24.303122 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:24.302843    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd" event={"ID":"c6646064-e1b1-4aef-ba29-bc599e34a0fa","Type":"ContainerDied","Data":"f7cd5504008523d802d056d65dec3e0df9ae90b6bf23a51a57e738adad7ce5d8"}
Sep 16 18:50:31.334287 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:31.334238    2584 generic.go:356] "Generic (PLEG): container finished" podID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerID="994b2e3abb25dd4a1964fe138b630b2e34cc69bd9329398feb174dd6f1d60178" exitCode=0
Sep 16 18:50:31.334658 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:31.334297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd" event={"ID":"c6646064-e1b1-4aef-ba29-bc599e34a0fa","Type":"ContainerDied","Data":"994b2e3abb25dd4a1964fe138b630b2e34cc69bd9329398feb174dd6f1d60178"}
Sep 16 18:50:32.508050 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.508024    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:32.621591 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.621523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm7rp\" (UniqueName: \"kubernetes.io/projected/c6646064-e1b1-4aef-ba29-bc599e34a0fa-kube-api-access-lm7rp\") pod \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") "
Sep 16 18:50:32.621709 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.621594    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-bundle\") pod \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") "
Sep 16 18:50:32.621709 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.621622    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-util\") pod \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\" (UID: \"c6646064-e1b1-4aef-ba29-bc599e34a0fa\") "
Sep 16 18:50:32.622213 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.622185    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-bundle" (OuterVolumeSpecName: "bundle") pod "c6646064-e1b1-4aef-ba29-bc599e34a0fa" (UID: "c6646064-e1b1-4aef-ba29-bc599e34a0fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:32.623792 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.623762    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6646064-e1b1-4aef-ba29-bc599e34a0fa-kube-api-access-lm7rp" (OuterVolumeSpecName: "kube-api-access-lm7rp") pod "c6646064-e1b1-4aef-ba29-bc599e34a0fa" (UID: "c6646064-e1b1-4aef-ba29-bc599e34a0fa"). InnerVolumeSpecName "kube-api-access-lm7rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:32.625705 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.625679    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-util" (OuterVolumeSpecName: "util") pod "c6646064-e1b1-4aef-ba29-bc599e34a0fa" (UID: "c6646064-e1b1-4aef-ba29-bc599e34a0fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:32.722105 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.722083    2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-bundle\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:32.722105 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.722103    2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6646064-e1b1-4aef-ba29-bc599e34a0fa-util\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:32.722233 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:32.722115    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm7rp\" (UniqueName: \"kubernetes.io/projected/c6646064-e1b1-4aef-ba29-bc599e34a0fa-kube-api-access-lm7rp\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:50:33.343570 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:33.343541    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd"
Sep 16 18:50:33.343723 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:33.343538    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4ktvd" event={"ID":"c6646064-e1b1-4aef-ba29-bc599e34a0fa","Type":"ContainerDied","Data":"9b7f14d13fe5f7485da304325d08c8c05735b3571901d028477af4aa0538bc1d"}
Sep 16 18:50:33.343723 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:33.343656    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b7f14d13fe5f7485da304325d08c8c05735b3571901d028477af4aa0538bc1d"
Sep 16 18:50:36.676445 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676412    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"]
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676688    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerName="extract"
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676698    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerName="extract"
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676707    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerName="util"
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676712    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerName="util"
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676722    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="448a802b-3b71-4347-ba1c-4bc90364f3a2" containerName="console"
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676728    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="448a802b-3b71-4347-ba1c-4bc90364f3a2" containerName="console"
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676737    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerName="pull"
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676742    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerName="pull"
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676784    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="448a802b-3b71-4347-ba1c-4bc90364f3a2" containerName="console"
Sep 16 18:50:36.676876 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.676791    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6646064-e1b1-4aef-ba29-bc599e34a0fa" containerName="extract"
Sep 16 18:50:36.679404 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.679388    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:50:36.682571 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.682539    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:50:36.682708 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.682668    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:50:36.682884 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.682869    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-m7flc\""
Sep 16 18:50:36.682961 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.682939    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:50:36.695692 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.695671    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"]
Sep 16 18:50:36.747696 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.747673    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdqdj\" (UniqueName: \"kubernetes.io/projected/665ba801-573c-4a08-a3d9-9db30b442a4b-kube-api-access-bdqdj\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z2w9l\" (UID: \"665ba801-573c-4a08-a3d9-9db30b442a4b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:50:36.747806 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.747711    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/665ba801-573c-4a08-a3d9-9db30b442a4b-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z2w9l\" (UID: \"665ba801-573c-4a08-a3d9-9db30b442a4b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:50:36.848063 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.848041    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/665ba801-573c-4a08-a3d9-9db30b442a4b-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z2w9l\" (UID: \"665ba801-573c-4a08-a3d9-9db30b442a4b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:50:36.848154 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.848102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdqdj\" (UniqueName: \"kubernetes.io/projected/665ba801-573c-4a08-a3d9-9db30b442a4b-kube-api-access-bdqdj\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z2w9l\" (UID: \"665ba801-573c-4a08-a3d9-9db30b442a4b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:50:36.850314 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.850295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/665ba801-573c-4a08-a3d9-9db30b442a4b-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z2w9l\" (UID: \"665ba801-573c-4a08-a3d9-9db30b442a4b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:50:36.859219 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.859191    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdqdj\" (UniqueName: \"kubernetes.io/projected/665ba801-573c-4a08-a3d9-9db30b442a4b-kube-api-access-bdqdj\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z2w9l\" (UID: \"665ba801-573c-4a08-a3d9-9db30b442a4b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:50:36.989135 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:36.989071    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:50:37.266255 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:37.266231    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"]
Sep 16 18:50:37.268478 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:50:37.268451    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665ba801_573c_4a08_a3d9_9db30b442a4b.slice/crio-06d17eb215d5644890fcbefd28056be87e6d79291027ba93814caede3b21a8ea WatchSource:0}: Error finding container 06d17eb215d5644890fcbefd28056be87e6d79291027ba93814caede3b21a8ea: Status 404 returned error can't find the container with id 06d17eb215d5644890fcbefd28056be87e6d79291027ba93814caede3b21a8ea
Sep 16 18:50:37.361325 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:37.361292    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l" event={"ID":"665ba801-573c-4a08-a3d9-9db30b442a4b","Type":"ContainerStarted","Data":"06d17eb215d5644890fcbefd28056be87e6d79291027ba93814caede3b21a8ea"}
Sep 16 18:50:41.205748 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.205715    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2f9l6"]
Sep 16 18:50:41.208942 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.208926    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.213142 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.213120    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:50:41.213240 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.213129    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tmm7t\""
Sep 16 18:50:41.213387 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.213373    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:50:41.229356 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.229331    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2f9l6"]
Sep 16 18:50:41.282948 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.282921    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b63609e1-72cd-4e10-8317-bfdfba652689-cabundle0\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.283087 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.282978    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwmjt\" (UniqueName: \"kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-kube-api-access-pwmjt\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.283160 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.283095    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-certificates\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.379612 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.379580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l" event={"ID":"665ba801-573c-4a08-a3d9-9db30b442a4b","Type":"ContainerStarted","Data":"de2cfc245ea0cbb8d2e1b86d20ba5610bbd83c741a14072a3b04254ffd49409d"}
Sep 16 18:50:41.379740 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.379681    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:50:41.383722 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.383693    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-certificates\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.383852 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.383763    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b63609e1-72cd-4e10-8317-bfdfba652689-cabundle0\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.383852 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.383817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmjt\" (UniqueName: \"kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-kube-api-access-pwmjt\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.383972 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:41.383858    2584 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 18:50:41.383972 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:41.383878    2584 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:50:41.383972 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:41.383890    2584 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:50:41.383972 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:41.383905    2584 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-2f9l6: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:50:41.383972 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:41.383971    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-certificates podName:b63609e1-72cd-4e10-8317-bfdfba652689 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:41.883954 +0000 UTC m=+372.550893130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-certificates") pod "keda-operator-579c8958f7-2f9l6" (UID: "b63609e1-72cd-4e10-8317-bfdfba652689") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:50:41.384427 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.384410    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b63609e1-72cd-4e10-8317-bfdfba652689-cabundle0\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.392916 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.392895    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmjt\" (UniqueName: \"kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-kube-api-access-pwmjt\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.403871 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.403821    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l" podStartSLOduration=2.27642479 podStartE2EDuration="5.40380401s" totalImagesPullingTime="3.12737922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:41.401822787 +0000 UTC m=+372.068761926" watchObservedRunningTime="2026-09-16 18:50:41.40380401 +0000 UTC m=+372.070743161"
Sep 16 18:50:41.887799 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:41.887768    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-certificates\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:41.887956 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:41.887910    2584 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:50:41.887956 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:41.887924    2584 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:50:41.887956 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:41.887932    2584 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-2f9l6: references non-existent secret key: ca.crt
Sep 16 18:50:41.888107 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:50:41.887985    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-certificates podName:b63609e1-72cd-4e10-8317-bfdfba652689 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:42.887968673 +0000 UTC m=+373.554907807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-certificates") pod "keda-operator-579c8958f7-2f9l6" (UID: "b63609e1-72cd-4e10-8317-bfdfba652689") : references non-existent secret key: ca.crt
Sep 16 18:50:42.895794 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:42.895759    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-certificates\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:42.898154 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:42.898135    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b63609e1-72cd-4e10-8317-bfdfba652689-certificates\") pod \"keda-operator-579c8958f7-2f9l6\" (UID: \"b63609e1-72cd-4e10-8317-bfdfba652689\") " pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:43.021018 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:43.020989    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:43.315849 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:50:43.315820    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63609e1_72cd_4e10_8317_bfdfba652689.slice/crio-6c85b1dd552b99af901281fe852f8d2533c45c369a2cffcb5fad82fe9f9590f8 WatchSource:0}: Error finding container 6c85b1dd552b99af901281fe852f8d2533c45c369a2cffcb5fad82fe9f9590f8: Status 404 returned error can't find the container with id 6c85b1dd552b99af901281fe852f8d2533c45c369a2cffcb5fad82fe9f9590f8
Sep 16 18:50:43.343708 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:43.343682    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2f9l6"]
Sep 16 18:50:43.388569 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:43.388540    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-2f9l6" event={"ID":"b63609e1-72cd-4e10-8317-bfdfba652689","Type":"ContainerStarted","Data":"6c85b1dd552b99af901281fe852f8d2533c45c369a2cffcb5fad82fe9f9590f8"}
Sep 16 18:50:46.403093 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:46.403015    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-2f9l6" event={"ID":"b63609e1-72cd-4e10-8317-bfdfba652689","Type":"ContainerStarted","Data":"addccdfca1fbfed22189606d7d1458bf89f9548a51f4031764a1a7faff2c0cd1"}
Sep 16 18:50:46.403454 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:46.403142    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:50:46.421261 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:50:46.421216    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-2f9l6" podStartSLOduration=2.707324253 podStartE2EDuration="5.421204232s" totalImagesPullingTime="2.713879979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:46.419896643 +0000 UTC m=+377.086835789" watchObservedRunningTime="2026-09-16 18:50:46.421204232 +0000 UTC m=+377.088143378"
Sep 16 18:51:02.385964 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:51:02.385927    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z2w9l"
Sep 16 18:51:07.409849 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:51:07.409801    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-2f9l6"
Sep 16 18:52:03.879780 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:03.879745    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-2s9x6"]
Sep 16 18:52:03.883072 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:03.883051    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:03.886323 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:03.886303    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:52:03.887846 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:03.887817    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-q8wzg\""
Sep 16 18:52:03.888078 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:03.888060    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 18:52:03.893217 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:03.893193    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:52:03.916002 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:03.915974    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-2s9x6"]
Sep 16 18:52:03.994642 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:03.994615    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-cert\") pod \"kserve-controller-manager-75c4c6c467-2s9x6\" (UID: \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\") " pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:03.994756 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:03.994711    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqdbr\" (UniqueName: \"kubernetes.io/projected/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-kube-api-access-zqdbr\") pod \"kserve-controller-manager-75c4c6c467-2s9x6\" (UID: \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\") " pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:04.005134 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.005113    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-rm98n"]
Sep 16 18:52:04.008286 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.008258    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:04.022251 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.022232    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:52:04.022695 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.022678    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vwxwg\""
Sep 16 18:52:04.038542 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.038519    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rm98n"]
Sep 16 18:52:04.095565 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.095541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqdbr\" (UniqueName: \"kubernetes.io/projected/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-kube-api-access-zqdbr\") pod \"kserve-controller-manager-75c4c6c467-2s9x6\" (UID: \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\") " pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:04.095681 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.095588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-cert\") pod \"kserve-controller-manager-75c4c6c467-2s9x6\" (UID: \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\") " pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:04.098013 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.097989    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-cert\") pod \"kserve-controller-manager-75c4c6c467-2s9x6\" (UID: \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\") " pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:04.113009 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.112984    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqdbr\" (UniqueName: \"kubernetes.io/projected/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-kube-api-access-zqdbr\") pod \"kserve-controller-manager-75c4c6c467-2s9x6\" (UID: \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\") " pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:04.196589 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.196567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e36f7ba-0e09-4f8b-9b9b-ad936017dd95-data\") pod \"seaweedfs-64568bcd49-rm98n\" (UID: \"9e36f7ba-0e09-4f8b-9b9b-ad936017dd95\") " pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:04.196678 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.196604    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzsb\" (UniqueName: \"kubernetes.io/projected/9e36f7ba-0e09-4f8b-9b9b-ad936017dd95-kube-api-access-4wzsb\") pod \"seaweedfs-64568bcd49-rm98n\" (UID: \"9e36f7ba-0e09-4f8b-9b9b-ad936017dd95\") " pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:04.200570 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.200550    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:04.297790 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.297760    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e36f7ba-0e09-4f8b-9b9b-ad936017dd95-data\") pod \"seaweedfs-64568bcd49-rm98n\" (UID: \"9e36f7ba-0e09-4f8b-9b9b-ad936017dd95\") " pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:04.297938 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.297797    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzsb\" (UniqueName: \"kubernetes.io/projected/9e36f7ba-0e09-4f8b-9b9b-ad936017dd95-kube-api-access-4wzsb\") pod \"seaweedfs-64568bcd49-rm98n\" (UID: \"9e36f7ba-0e09-4f8b-9b9b-ad936017dd95\") " pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:04.298179 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.298154    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e36f7ba-0e09-4f8b-9b9b-ad936017dd95-data\") pod \"seaweedfs-64568bcd49-rm98n\" (UID: \"9e36f7ba-0e09-4f8b-9b9b-ad936017dd95\") " pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:04.309351 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.309323    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzsb\" (UniqueName: \"kubernetes.io/projected/9e36f7ba-0e09-4f8b-9b9b-ad936017dd95-kube-api-access-4wzsb\") pod \"seaweedfs-64568bcd49-rm98n\" (UID: \"9e36f7ba-0e09-4f8b-9b9b-ad936017dd95\") " pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:04.319020 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.318975    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:04.480607 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.480540    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-2s9x6"]
Sep 16 18:52:04.483396 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:52:04.483372    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc2f98d_4b52_48ee_8d9f_d431b7d3a809.slice/crio-bbaccbe94dc427a309e9065a4272c5a65c56c460f92daf5a1bf99def4308aa24 WatchSource:0}: Error finding container bbaccbe94dc427a309e9065a4272c5a65c56c460f92daf5a1bf99def4308aa24: Status 404 returned error can't find the container with id bbaccbe94dc427a309e9065a4272c5a65c56c460f92daf5a1bf99def4308aa24
Sep 16 18:52:04.589875 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.589848    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rm98n"]
Sep 16 18:52:04.592699 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:52:04.592672    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e36f7ba_0e09_4f8b_9b9b_ad936017dd95.slice/crio-316ff641db3a31ef2a1e1391a46af1305a40440a28d1106af76aff9c09891d13 WatchSource:0}: Error finding container 316ff641db3a31ef2a1e1391a46af1305a40440a28d1106af76aff9c09891d13: Status 404 returned error can't find the container with id 316ff641db3a31ef2a1e1391a46af1305a40440a28d1106af76aff9c09891d13
Sep 16 18:52:04.726586 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.726561    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rm98n" event={"ID":"9e36f7ba-0e09-4f8b-9b9b-ad936017dd95","Type":"ContainerStarted","Data":"316ff641db3a31ef2a1e1391a46af1305a40440a28d1106af76aff9c09891d13"}
Sep 16 18:52:04.727616 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:04.727593    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6" event={"ID":"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809","Type":"ContainerStarted","Data":"bbaccbe94dc427a309e9065a4272c5a65c56c460f92daf5a1bf99def4308aa24"}
Sep 16 18:52:08.754503 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:08.754452    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rm98n" event={"ID":"9e36f7ba-0e09-4f8b-9b9b-ad936017dd95","Type":"ContainerStarted","Data":"d354ce1a2375d393c4e15f046cb3441bbe958eab7de1fd432f7da7ad8be40409"}
Sep 16 18:52:08.754919 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:08.754557    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:08.755994 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:08.755968    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6" event={"ID":"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809","Type":"ContainerStarted","Data":"1b5b9ec92a6c8c87bb7f80b378f1b76008eb47eb54360964d083d5e727edb33b"}
Sep 16 18:52:08.756094 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:08.756076    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:08.775292 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:08.775240    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-rm98n" podStartSLOduration=2.176328489 podStartE2EDuration="5.775228249s" totalImagesPullingTime="3.59889976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:08.774676049 +0000 UTC m=+459.441615215" watchObservedRunningTime="2026-09-16 18:52:08.775228249 +0000 UTC m=+459.442167389"
Sep 16 18:52:08.795611 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:08.795568    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6" podStartSLOduration=2.143522786 podStartE2EDuration="5.795552896s" totalImagesPullingTime="3.65203011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:08.794381357 +0000 UTC m=+459.461320523" watchObservedRunningTime="2026-09-16 18:52:08.795552896 +0000 UTC m=+459.462492042"
Sep 16 18:52:14.762139 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:14.762110    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-rm98n"
Sep 16 18:52:39.765107 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:39.765079    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:49.725798 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.725762    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-2s9x6"]
Sep 16 18:52:49.726352 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.726034    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6" podUID="1bc2f98d-4b52-48ee-8d9f-d431b7d3a809" containerName="manager" containerID="cri-o://1b5b9ec92a6c8c87bb7f80b378f1b76008eb47eb54360964d083d5e727edb33b" gracePeriod=10
Sep 16 18:52:49.754326 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.754301    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-t8c2x"]
Sep 16 18:52:49.757494 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.757478    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:52:49.760160 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.760125    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6" podUID="1bc2f98d-4b52-48ee-8d9f-d431b7d3a809" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": dial tcp 10.133.0.28:8081: connect: connection refused"
Sep 16 18:52:49.770343 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.770321    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-t8c2x"]
Sep 16 18:52:49.813901 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.813870    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/998302ad-889b-430c-afa8-a60710b4533d-cert\") pod \"kserve-controller-manager-75c4c6c467-t8c2x\" (UID: \"998302ad-889b-430c-afa8-a60710b4533d\") " pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:52:49.814041 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.813980    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2q9g\" (UniqueName: \"kubernetes.io/projected/998302ad-889b-430c-afa8-a60710b4533d-kube-api-access-c2q9g\") pod \"kserve-controller-manager-75c4c6c467-t8c2x\" (UID: \"998302ad-889b-430c-afa8-a60710b4533d\") " pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:52:49.914415 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.914382    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2q9g\" (UniqueName: \"kubernetes.io/projected/998302ad-889b-430c-afa8-a60710b4533d-kube-api-access-c2q9g\") pod \"kserve-controller-manager-75c4c6c467-t8c2x\" (UID: \"998302ad-889b-430c-afa8-a60710b4533d\") " pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:52:49.914573 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.914437    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/998302ad-889b-430c-afa8-a60710b4533d-cert\") pod \"kserve-controller-manager-75c4c6c467-t8c2x\" (UID: \"998302ad-889b-430c-afa8-a60710b4533d\") " pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:52:49.917156 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.917122    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/998302ad-889b-430c-afa8-a60710b4533d-cert\") pod \"kserve-controller-manager-75c4c6c467-t8c2x\" (UID: \"998302ad-889b-430c-afa8-a60710b4533d\") " pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:52:49.923802 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.923772    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2q9g\" (UniqueName: \"kubernetes.io/projected/998302ad-889b-430c-afa8-a60710b4533d-kube-api-access-c2q9g\") pod \"kserve-controller-manager-75c4c6c467-t8c2x\" (UID: \"998302ad-889b-430c-afa8-a60710b4533d\") " pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:52:49.931464 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.931436    2584 generic.go:356] "Generic (PLEG): container finished" podID="1bc2f98d-4b52-48ee-8d9f-d431b7d3a809" containerID="1b5b9ec92a6c8c87bb7f80b378f1b76008eb47eb54360964d083d5e727edb33b" exitCode=0
Sep 16 18:52:49.931592 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.931511    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6" event={"ID":"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809","Type":"ContainerDied","Data":"1b5b9ec92a6c8c87bb7f80b378f1b76008eb47eb54360964d083d5e727edb33b"}
Sep 16 18:52:49.963365 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:49.963340    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:50.015721 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.015630    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-cert\") pod \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\" (UID: \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\") "
Sep 16 18:52:50.015721 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.015698    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqdbr\" (UniqueName: \"kubernetes.io/projected/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-kube-api-access-zqdbr\") pod \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\" (UID: \"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809\") "
Sep 16 18:52:50.017746 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.017711    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-cert" (OuterVolumeSpecName: "cert") pod "1bc2f98d-4b52-48ee-8d9f-d431b7d3a809" (UID: "1bc2f98d-4b52-48ee-8d9f-d431b7d3a809"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:50.017869 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.017775    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-kube-api-access-zqdbr" (OuterVolumeSpecName: "kube-api-access-zqdbr") pod "1bc2f98d-4b52-48ee-8d9f-d431b7d3a809" (UID: "1bc2f98d-4b52-48ee-8d9f-d431b7d3a809"). InnerVolumeSpecName "kube-api-access-zqdbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:50.107276 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.107234    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:52:50.116556 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.116534    2584 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:52:50.116556 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.116556    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqdbr\" (UniqueName: \"kubernetes.io/projected/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809-kube-api-access-zqdbr\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 18:52:50.380544 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.380498    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-t8c2x"]
Sep 16 18:52:50.382718 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:52:50.382690    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998302ad_889b_430c_afa8_a60710b4533d.slice/crio-8bc2c21171da9a2b0dc7f41d961571ae96d97cd2d9fab659e190728c703599c2 WatchSource:0}: Error finding container 8bc2c21171da9a2b0dc7f41d961571ae96d97cd2d9fab659e190728c703599c2: Status 404 returned error can't find the container with id 8bc2c21171da9a2b0dc7f41d961571ae96d97cd2d9fab659e190728c703599c2
Sep 16 18:52:50.937655 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.937623    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6"
Sep 16 18:52:50.938465 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.937630    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-2s9x6" event={"ID":"1bc2f98d-4b52-48ee-8d9f-d431b7d3a809","Type":"ContainerDied","Data":"bbaccbe94dc427a309e9065a4272c5a65c56c460f92daf5a1bf99def4308aa24"}
Sep 16 18:52:50.938465 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.937778    2584 scope.go:117] "RemoveContainer" containerID="1b5b9ec92a6c8c87bb7f80b378f1b76008eb47eb54360964d083d5e727edb33b"
Sep 16 18:52:50.939775 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.939749    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x" event={"ID":"998302ad-889b-430c-afa8-a60710b4533d","Type":"ContainerStarted","Data":"8f03fdc85c63d60763bb97f88bfb5e4bced2a03f56b40a3edeb4f4581b282962"}
Sep 16 18:52:50.939873 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.939806    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:52:50.939873 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.939834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x" event={"ID":"998302ad-889b-430c-afa8-a60710b4533d","Type":"ContainerStarted","Data":"8bc2c21171da9a2b0dc7f41d961571ae96d97cd2d9fab659e190728c703599c2"}
Sep 16 18:52:50.960783 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.960732    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x" podStartSLOduration=1.526378998 podStartE2EDuration="1.960715513s" totalImagesPullingTime="434.336515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:50.959435565 +0000 UTC m=+501.626374729" watchObservedRunningTime="2026-09-16 18:52:50.960715513 +0000 UTC m=+501.627654660"
Sep 16 18:52:50.974559 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.974534    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-2s9x6"]
Sep 16 18:52:50.979573 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:50.979553    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-2s9x6"]
Sep 16 18:52:51.819444 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:52:51.817381    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bc2f98d-4b52-48ee-8d9f-d431b7d3a809" path="/var/lib/kubelet/pods/1bc2f98d-4b52-48ee-8d9f-d431b7d3a809/volumes"
Sep 16 18:53:21.947860 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:21.947831    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-75c4c6c467-t8c2x"
Sep 16 18:53:23.245121 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.245087    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-dpx5t"]
Sep 16 18:53:23.245568 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.245411    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bc2f98d-4b52-48ee-8d9f-d431b7d3a809" containerName="manager"
Sep 16 18:53:23.245568 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.245424    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bc2f98d-4b52-48ee-8d9f-d431b7d3a809" containerName="manager"
Sep 16 18:53:23.245568 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.245482    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bc2f98d-4b52-48ee-8d9f-d431b7d3a809" containerName="manager"
Sep 16 18:53:23.248228 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.248209    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:53:23.250690 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.250666    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:53:23.250818 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.250671    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-6796r\""
Sep 16 18:53:23.260100 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.260075    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dpx5t"]
Sep 16 18:53:23.264521 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.264499    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-8x9gn"]
Sep 16 18:53:23.267920 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.267899    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.270975 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.270698    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:53:23.270975 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.270751    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:53:23.271397 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.271382    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-ltcf7\""
Sep 16 18:53:23.281905 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.281882    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-8x9gn"]
Sep 16 18:53:23.359957 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.359925    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89x94\" (UniqueName: \"kubernetes.io/projected/24075ac7-eed0-47f9-ac7e-227aa61530c2-kube-api-access-89x94\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.360114 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.359973    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pqcd\" (UniqueName: \"kubernetes.io/projected/8211dde6-b168-4135-a0c4-c4b9a4b28f08-kube-api-access-6pqcd\") pod \"model-serving-api-7b954ffc95-dpx5t\" (UID: \"8211dde6-b168-4135-a0c4-c4b9a4b28f08\") " pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:53:23.360114 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.360079    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24075ac7-eed0-47f9-ac7e-227aa61530c2-cert\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.360114 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.360108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24075ac7-eed0-47f9-ac7e-227aa61530c2-metrics-certs\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.360234 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.360171    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8211dde6-b168-4135-a0c4-c4b9a4b28f08-tls-certs\") pod \"model-serving-api-7b954ffc95-dpx5t\" (UID: \"8211dde6-b168-4135-a0c4-c4b9a4b28f08\") " pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:53:23.460502 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.460476    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8211dde6-b168-4135-a0c4-c4b9a4b28f08-tls-certs\") pod \"model-serving-api-7b954ffc95-dpx5t\" (UID: \"8211dde6-b168-4135-a0c4-c4b9a4b28f08\") " pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:53:23.460615 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.460510    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89x94\" (UniqueName: \"kubernetes.io/projected/24075ac7-eed0-47f9-ac7e-227aa61530c2-kube-api-access-89x94\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.460615 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.460544    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pqcd\" (UniqueName: \"kubernetes.io/projected/8211dde6-b168-4135-a0c4-c4b9a4b28f08-kube-api-access-6pqcd\") pod \"model-serving-api-7b954ffc95-dpx5t\" (UID: \"8211dde6-b168-4135-a0c4-c4b9a4b28f08\") " pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:53:23.460615 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.460588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24075ac7-eed0-47f9-ac7e-227aa61530c2-cert\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.460735 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.460614    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24075ac7-eed0-47f9-ac7e-227aa61530c2-metrics-certs\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.460735 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:53:23.460719    2584 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:53:23.460815 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:53:23.460794    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24075ac7-eed0-47f9-ac7e-227aa61530c2-cert podName:24075ac7-eed0-47f9-ac7e-227aa61530c2 nodeName:}" failed. No retries permitted until 2026-09-16 18:53:23.960773057 +0000 UTC m=+534.627712189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/24075ac7-eed0-47f9-ac7e-227aa61530c2-cert") pod "odh-model-controller-7b966d7984-8x9gn" (UID: "24075ac7-eed0-47f9-ac7e-227aa61530c2") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:53:23.462993 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.462949    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8211dde6-b168-4135-a0c4-c4b9a4b28f08-tls-certs\") pod \"model-serving-api-7b954ffc95-dpx5t\" (UID: \"8211dde6-b168-4135-a0c4-c4b9a4b28f08\") " pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:53:23.463230 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.463210    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24075ac7-eed0-47f9-ac7e-227aa61530c2-metrics-certs\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.471644 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.471624    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pqcd\" (UniqueName: \"kubernetes.io/projected/8211dde6-b168-4135-a0c4-c4b9a4b28f08-kube-api-access-6pqcd\") pod \"model-serving-api-7b954ffc95-dpx5t\" (UID: \"8211dde6-b168-4135-a0c4-c4b9a4b28f08\") " pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:53:23.472249 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.472227    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89x94\" (UniqueName: \"kubernetes.io/projected/24075ac7-eed0-47f9-ac7e-227aa61530c2-kube-api-access-89x94\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.559143 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.559088    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:53:23.829433 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.829387    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dpx5t"]
Sep 16 18:53:23.831577 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:53:23.831551    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8211dde6_b168_4135_a0c4_c4b9a4b28f08.slice/crio-bb110edb7eec03178d2217109c4ec5529389516801956adc869395e72f709361 WatchSource:0}: Error finding container bb110edb7eec03178d2217109c4ec5529389516801956adc869395e72f709361: Status 404 returned error can't find the container with id bb110edb7eec03178d2217109c4ec5529389516801956adc869395e72f709361
Sep 16 18:53:23.964185 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.964158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24075ac7-eed0-47f9-ac7e-227aa61530c2-cert\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:23.966393 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:23.966374    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24075ac7-eed0-47f9-ac7e-227aa61530c2-cert\") pod \"odh-model-controller-7b966d7984-8x9gn\" (UID: \"24075ac7-eed0-47f9-ac7e-227aa61530c2\") " pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:24.079384 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:24.079327    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dpx5t" event={"ID":"8211dde6-b168-4135-a0c4-c4b9a4b28f08","Type":"ContainerStarted","Data":"bb110edb7eec03178d2217109c4ec5529389516801956adc869395e72f709361"}
Sep 16 18:53:24.179475 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:24.179453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:24.452314 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:24.452283    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-8x9gn"]
Sep 16 18:53:25.087495 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:25.087436    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-8x9gn" event={"ID":"24075ac7-eed0-47f9-ac7e-227aa61530c2","Type":"ContainerStarted","Data":"3ebd3a8b5d7b050cc3d782341840fd88d49e2e030605d1b59aeb65fe76462884"}
Sep 16 18:53:28.104507 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:28.104466    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-8x9gn" event={"ID":"24075ac7-eed0-47f9-ac7e-227aa61530c2","Type":"ContainerStarted","Data":"ea7e871550cf0ad209ba631ca558851903e0b44d8a4c9ae5dba0912803b470e7"}
Sep 16 18:53:28.104950 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:28.104567    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:28.105960 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:28.105934    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dpx5t" event={"ID":"8211dde6-b168-4135-a0c4-c4b9a4b28f08","Type":"ContainerStarted","Data":"5baebb9a293ad81baadab923037786470e06772affd052b7610ceae104634aff"}
Sep 16 18:53:28.106092 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:28.106076    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:53:28.124027 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:28.123982    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-8x9gn" podStartSLOduration=2.111103128 podStartE2EDuration="5.123969584s" totalImagesPullingTime="3.012866456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:28.121987729 +0000 UTC m=+538.788926895" watchObservedRunningTime="2026-09-16 18:53:28.123969584 +0000 UTC m=+538.790908729"
Sep 16 18:53:28.143518 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:28.143479    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-dpx5t" podStartSLOduration=1.552422739 podStartE2EDuration="5.143467051s" totalImagesPullingTime="3.591044312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:28.141852134 +0000 UTC m=+538.808791298" watchObservedRunningTime="2026-09-16 18:53:28.143467051 +0000 UTC m=+538.810406197"
Sep 16 18:53:39.112166 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:39.112137    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-8x9gn"
Sep 16 18:53:39.114780 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:53:39.114759    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-dpx5t"
Sep 16 18:55:33.205285 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.205238    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"]
Sep 16 18:55:33.208670 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.208646    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.211331 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.211309    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:55:33.211471 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.211357    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 16 18:55:33.216694 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.216670    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"]
Sep 16 18:55:33.254451 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.254409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/96b04e12-4ee6-4e95-b400-a33a933009e8-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-jkz2w\" (UID: \"96b04e12-4ee6-4e95-b400-a33a933009e8\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.254649 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.254518    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/96b04e12-4ee6-4e95-b400-a33a933009e8-data\") pod \"seaweedfs-tls-custom-8b44d767d-jkz2w\" (UID: \"96b04e12-4ee6-4e95-b400-a33a933009e8\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.254649 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.254546    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7xs6\" (UniqueName: \"kubernetes.io/projected/96b04e12-4ee6-4e95-b400-a33a933009e8-kube-api-access-d7xs6\") pod \"seaweedfs-tls-custom-8b44d767d-jkz2w\" (UID: \"96b04e12-4ee6-4e95-b400-a33a933009e8\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.355568 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.355523    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/96b04e12-4ee6-4e95-b400-a33a933009e8-data\") pod \"seaweedfs-tls-custom-8b44d767d-jkz2w\" (UID: \"96b04e12-4ee6-4e95-b400-a33a933009e8\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.355568 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.355568    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7xs6\" (UniqueName: \"kubernetes.io/projected/96b04e12-4ee6-4e95-b400-a33a933009e8-kube-api-access-d7xs6\") pod \"seaweedfs-tls-custom-8b44d767d-jkz2w\" (UID: \"96b04e12-4ee6-4e95-b400-a33a933009e8\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.355839 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.355624    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/96b04e12-4ee6-4e95-b400-a33a933009e8-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-jkz2w\" (UID: \"96b04e12-4ee6-4e95-b400-a33a933009e8\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.356003 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.355978    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/96b04e12-4ee6-4e95-b400-a33a933009e8-data\") pod \"seaweedfs-tls-custom-8b44d767d-jkz2w\" (UID: \"96b04e12-4ee6-4e95-b400-a33a933009e8\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.358188 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.358168    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/96b04e12-4ee6-4e95-b400-a33a933009e8-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-jkz2w\" (UID: \"96b04e12-4ee6-4e95-b400-a33a933009e8\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.364652 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.364627    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7xs6\" (UniqueName: \"kubernetes.io/projected/96b04e12-4ee6-4e95-b400-a33a933009e8-kube-api-access-d7xs6\") pod \"seaweedfs-tls-custom-8b44d767d-jkz2w\" (UID: \"96b04e12-4ee6-4e95-b400-a33a933009e8\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.519031 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.518941    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"
Sep 16 18:55:33.792522 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.792496    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-jkz2w"]
Sep 16 18:55:33.795243 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:55:33.795217    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b04e12_4ee6_4e95_b400_a33a933009e8.slice/crio-016cc61ee0cf6da316028f38432dd9bdb8c704759948b1cecb51eba4f6c473c6 WatchSource:0}: Error finding container 016cc61ee0cf6da316028f38432dd9bdb8c704759948b1cecb51eba4f6c473c6: Status 404 returned error can't find the container with id 016cc61ee0cf6da316028f38432dd9bdb8c704759948b1cecb51eba4f6c473c6
Sep 16 18:55:33.796588 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:33.796572    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:55:34.654527 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:34.654485    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w" event={"ID":"96b04e12-4ee6-4e95-b400-a33a933009e8","Type":"ContainerStarted","Data":"0a5f7bfd17310a260a73361e223474b2296c0c05dad1ad960cd4b25b5234baed"}
Sep 16 18:55:34.654900 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:34.654580    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w" event={"ID":"96b04e12-4ee6-4e95-b400-a33a933009e8","Type":"ContainerStarted","Data":"016cc61ee0cf6da316028f38432dd9bdb8c704759948b1cecb51eba4f6c473c6"}
Sep 16 18:55:34.672456 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:55:34.672402    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-jkz2w" podStartSLOduration=1.419662929 podStartE2EDuration="1.672384686s" totalImagesPullingTime="252.721757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:34.670657238 +0000 UTC m=+665.337596402" watchObservedRunningTime="2026-09-16 18:55:34.672384686 +0000 UTC m=+665.339323832"
Sep 16 18:59:41.357866 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.357831    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"]
Sep 16 18:59:41.361254 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.361233    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:41.368391 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.368371    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:59:41.368516 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.368418    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 18:59:41.369337 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.369322    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 16 18:59:41.369526 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.369501    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:59:41.369806 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.369791    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 16 18:59:41.380447 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.380426    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"]
Sep 16 18:59:41.450330 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.450301    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:41.450449 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.450360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bmzp\" (UniqueName: \"kubernetes.io/projected/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-api-access-4bmzp\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:41.450449 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.450409    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:41.551434 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.551401    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bmzp\" (UniqueName: \"kubernetes.io/projected/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-api-access-4bmzp\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:41.551575 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.551458    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:41.551575 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.551484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:41.551707 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:59:41.551615    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 16 18:59:41.551707 ip-10-0-139-120 kubenswrapper[2584]: E0916 18:59:41.551691    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-proxy-tls podName:0ad3cf44-4463-4fb1-8e9c-9141f35ff949 nodeName:}" failed. No retries permitted until 2026-09-16 18:59:42.051669616 +0000 UTC m=+912.718608742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-qwz9q" (UID: "0ad3cf44-4463-4fb1-8e9c-9141f35ff949") : secret "message-dumper-predictor-serving-cert" not found
Sep 16 18:59:41.552100 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.552083    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:41.562258 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:41.562231    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bmzp\" (UniqueName: \"kubernetes.io/projected/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-api-access-4bmzp\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:42.056086 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:42.056052    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:42.058445 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:42.058421    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qwz9q\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:42.271927 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:42.271896    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:42.544317 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:42.544294    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"]
Sep 16 18:59:42.546442 ip-10-0-139-120 kubenswrapper[2584]: W0916 18:59:42.546416    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad3cf44_4463_4fb1_8e9c_9141f35ff949.slice/crio-22813cea4416f5d50a515fc81de4e1dd390e8828a97e84d4557d14eb5f5ebd3f WatchSource:0}: Error finding container 22813cea4416f5d50a515fc81de4e1dd390e8828a97e84d4557d14eb5f5ebd3f: Status 404 returned error can't find the container with id 22813cea4416f5d50a515fc81de4e1dd390e8828a97e84d4557d14eb5f5ebd3f
Sep 16 18:59:42.743709 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:42.743684    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" event={"ID":"0ad3cf44-4463-4fb1-8e9c-9141f35ff949","Type":"ContainerStarted","Data":"22813cea4416f5d50a515fc81de4e1dd390e8828a97e84d4557d14eb5f5ebd3f"}
Sep 16 18:59:44.756940 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:44.756850    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" event={"ID":"0ad3cf44-4463-4fb1-8e9c-9141f35ff949","Type":"ContainerStarted","Data":"a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a"}
Sep 16 18:59:46.768450 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:46.768412    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" event={"ID":"0ad3cf44-4463-4fb1-8e9c-9141f35ff949","Type":"ContainerStarted","Data":"ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8"}
Sep 16 18:59:46.768969 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:46.768632    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:46.768969 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:46.768685    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:46.770591 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:46.770569    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 18:59:46.789077 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:46.789033    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" podStartSLOduration=2.372069605 podStartE2EDuration="5.789021782s" totalImagesPullingTime="3.416952177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:46.787005157 +0000 UTC m=+917.453944323" watchObservedRunningTime="2026-09-16 18:59:46.789021782 +0000 UTC m=+917.455960928"
Sep 16 18:59:53.781836 ip-10-0-139-120 kubenswrapper[2584]: I0916 18:59:53.781806    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 19:01:36.426364 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:36.426333    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-qwz9q_0ad3cf44-4463-4fb1-8e9c-9141f35ff949/kserve-container/0.log"
Sep 16 19:01:36.630248 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:36.630199    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"]
Sep 16 19:01:36.630525 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:36.630491    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerName="kserve-container" containerID="cri-o://a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a" gracePeriod=30
Sep 16 19:01:36.630678 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:36.630530    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerName="kube-rbac-proxy" containerID="cri-o://ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8" gracePeriod=30
Sep 16 19:01:36.769239 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:36.769189    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 16 19:01:36.890105 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:36.890080    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 19:01:37.013004 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.012922    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bmzp\" (UniqueName: \"kubernetes.io/projected/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-api-access-4bmzp\") pod \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") "
Sep 16 19:01:37.013004 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.012994    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-rbac-proxy-sar-config\") pod \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") "
Sep 16 19:01:37.013199 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.013020    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-proxy-tls\") pod \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\" (UID: \"0ad3cf44-4463-4fb1-8e9c-9141f35ff949\") "
Sep 16 19:01:37.013367 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.013343    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ad3cf44-4463-4fb1-8e9c-9141f35ff949" (UID: "0ad3cf44-4463-4fb1-8e9c-9141f35ff949"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:37.015167 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.015146    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ad3cf44-4463-4fb1-8e9c-9141f35ff949" (UID: "0ad3cf44-4463-4fb1-8e9c-9141f35ff949"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:37.015295 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.015260    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-api-access-4bmzp" (OuterVolumeSpecName: "kube-api-access-4bmzp") pod "0ad3cf44-4463-4fb1-8e9c-9141f35ff949" (UID: "0ad3cf44-4463-4fb1-8e9c-9141f35ff949"). InnerVolumeSpecName "kube-api-access-4bmzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:37.113896 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.113857    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bmzp\" (UniqueName: \"kubernetes.io/projected/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-api-access-4bmzp\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:01:37.113896 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.113892    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:01:37.114072 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.113906    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ad3cf44-4463-4fb1-8e9c-9141f35ff949-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:01:37.255046 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.255014    2584 generic.go:356] "Generic (PLEG): container finished" podID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerID="ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8" exitCode=2
Sep 16 19:01:37.255046 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.255040    2584 generic.go:356] "Generic (PLEG): container finished" podID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerID="a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a" exitCode=2
Sep 16 19:01:37.255254 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.255065    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"
Sep 16 19:01:37.255254 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.255107    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" event={"ID":"0ad3cf44-4463-4fb1-8e9c-9141f35ff949","Type":"ContainerDied","Data":"ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8"}
Sep 16 19:01:37.255254 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.255172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" event={"ID":"0ad3cf44-4463-4fb1-8e9c-9141f35ff949","Type":"ContainerDied","Data":"a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a"}
Sep 16 19:01:37.255254 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.255196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q" event={"ID":"0ad3cf44-4463-4fb1-8e9c-9141f35ff949","Type":"ContainerDied","Data":"22813cea4416f5d50a515fc81de4e1dd390e8828a97e84d4557d14eb5f5ebd3f"}
Sep 16 19:01:37.255254 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.255227    2584 scope.go:117] "RemoveContainer" containerID="ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8"
Sep 16 19:01:37.263897 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.263882    2584 scope.go:117] "RemoveContainer" containerID="a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a"
Sep 16 19:01:37.271363 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.271346    2584 scope.go:117] "RemoveContainer" containerID="ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8"
Sep 16 19:01:37.271637 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:01:37.271620    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8\": container with ID starting with ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8 not found: ID does not exist" containerID="ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8"
Sep 16 19:01:37.271689 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.271659    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8"} err="failed to get container status \"ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8\": rpc error: code = NotFound desc = could not find container \"ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8\": container with ID starting with ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8 not found: ID does not exist"
Sep 16 19:01:37.271689 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.271675    2584 scope.go:117] "RemoveContainer" containerID="a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a"
Sep 16 19:01:37.271897 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:01:37.271883    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a\": container with ID starting with a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a not found: ID does not exist" containerID="a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a"
Sep 16 19:01:37.271933 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.271901    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a"} err="failed to get container status \"a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a\": rpc error: code = NotFound desc = could not find container \"a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a\": container with ID starting with a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a not found: ID does not exist"
Sep 16 19:01:37.271933 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.271912    2584 scope.go:117] "RemoveContainer" containerID="ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8"
Sep 16 19:01:37.272100 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.272082    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8"} err="failed to get container status \"ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8\": rpc error: code = NotFound desc = could not find container \"ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8\": container with ID starting with ab2fb15d41b6cb69780fe0002ebe5808d9c945b0614d00a9d0faace22c3f0ae8 not found: ID does not exist"
Sep 16 19:01:37.272138 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.272101    2584 scope.go:117] "RemoveContainer" containerID="a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a"
Sep 16 19:01:37.272317 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.272298    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a"} err="failed to get container status \"a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a\": rpc error: code = NotFound desc = could not find container \"a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a\": container with ID starting with a22cb2fef45359636f77fb17c3f68e9e7f7b51b47c778cc0ba6f171107734e4a not found: ID does not exist"
Sep 16 19:01:37.279742 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.279720    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"]
Sep 16 19:01:37.283685 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.283664    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qwz9q"]
Sep 16 19:01:37.816055 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:01:37.816024    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" path="/var/lib/kubelet/pods/0ad3cf44-4463-4fb1-8e9c-9141f35ff949/volumes"
Sep 16 19:02:12.887159 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:12.887098    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-865db6676c-zj9th"]
Sep 16 19:02:12.887634 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:12.887486    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerName="kube-rbac-proxy"
Sep 16 19:02:12.887634 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:12.887500    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerName="kube-rbac-proxy"
Sep 16 19:02:12.887634 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:12.887526    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerName="kserve-container"
Sep 16 19:02:12.887634 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:12.887532    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerName="kserve-container"
Sep 16 19:02:12.887634 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:12.887586    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerName="kube-rbac-proxy"
Sep 16 19:02:12.887634 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:12.887598    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad3cf44-4463-4fb1-8e9c-9141f35ff949" containerName="kserve-container"
Sep 16 19:02:12.890536 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:12.890518    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:12.909529 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:12.909502    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865db6676c-zj9th"]
Sep 16 19:02:13.003049 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.003014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f2ccc3-f370-41f8-a248-ae3d60243d64-console-serving-cert\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.003049 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.003057    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-console-config\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.003260 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.003087    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-oauth-serving-cert\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.003260 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.003164    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f2ccc3-f370-41f8-a248-ae3d60243d64-console-oauth-config\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.003260 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.003237    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-service-ca\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.003260 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.003256    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-trusted-ca-bundle\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.003414 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.003314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjpq\" (UniqueName: \"kubernetes.io/projected/19f2ccc3-f370-41f8-a248-ae3d60243d64-kube-api-access-ggjpq\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.104664 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.104625    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-service-ca\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.104664 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.104666    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-trusted-ca-bundle\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.104898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.104687    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjpq\" (UniqueName: \"kubernetes.io/projected/19f2ccc3-f370-41f8-a248-ae3d60243d64-kube-api-access-ggjpq\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.104898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.104721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f2ccc3-f370-41f8-a248-ae3d60243d64-console-serving-cert\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.104898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.104744    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-console-config\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.104898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.104766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-oauth-serving-cert\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.104898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.104793    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f2ccc3-f370-41f8-a248-ae3d60243d64-console-oauth-config\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.105518 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.105486    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-service-ca\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.105668 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.105645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-trusted-ca-bundle\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.105738 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.105642    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-oauth-serving-cert\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.105960 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.105940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f2ccc3-f370-41f8-a248-ae3d60243d64-console-config\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.107215 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.107197    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f2ccc3-f370-41f8-a248-ae3d60243d64-console-serving-cert\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.107335 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.107320    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f2ccc3-f370-41f8-a248-ae3d60243d64-console-oauth-config\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.113758 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.113726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjpq\" (UniqueName: \"kubernetes.io/projected/19f2ccc3-f370-41f8-a248-ae3d60243d64-kube-api-access-ggjpq\") pod \"console-865db6676c-zj9th\" (UID: \"19f2ccc3-f370-41f8-a248-ae3d60243d64\") " pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.199873 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.199839    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:13.477120 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:13.477095    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865db6676c-zj9th"]
Sep 16 19:02:13.479742 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:02:13.479714    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f2ccc3_f370_41f8_a248_ae3d60243d64.slice/crio-8f5598eac93d058bf4cc6b084d0395b90a69e9b48f431f95c57bd2889140bb68 WatchSource:0}: Error finding container 8f5598eac93d058bf4cc6b084d0395b90a69e9b48f431f95c57bd2889140bb68: Status 404 returned error can't find the container with id 8f5598eac93d058bf4cc6b084d0395b90a69e9b48f431f95c57bd2889140bb68
Sep 16 19:02:14.427116 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:14.427078    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865db6676c-zj9th" event={"ID":"19f2ccc3-f370-41f8-a248-ae3d60243d64","Type":"ContainerStarted","Data":"0b44a1536c8ec26db7da7adda9df6254182593831a60520bd80cb30b72bf9b3b"}
Sep 16 19:02:14.427514 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:14.427136    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865db6676c-zj9th" event={"ID":"19f2ccc3-f370-41f8-a248-ae3d60243d64","Type":"ContainerStarted","Data":"8f5598eac93d058bf4cc6b084d0395b90a69e9b48f431f95c57bd2889140bb68"}
Sep 16 19:02:14.449772 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:14.449722    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-865db6676c-zj9th" podStartSLOduration=2.449707993 podStartE2EDuration="2.449707993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:14.447377132 +0000 UTC m=+1065.114316298" watchObservedRunningTime="2026-09-16 19:02:14.449707993 +0000 UTC m=+1065.116647508"
Sep 16 19:02:23.200376 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:23.200295    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:23.200804 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:23.200385    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:23.205308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:23.205282    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:23.470354 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:23.470260    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-865db6676c-zj9th"
Sep 16 19:02:23.552376 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:23.552342    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b9bf4847-j9822"]
Sep 16 19:02:48.573509 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:48.573462    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59b9bf4847-j9822" podUID="5d335e37-5496-4c3d-bc25-a292e7b5865a" containerName="console" containerID="cri-o://63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b" gracePeriod=15
Sep 16 19:02:48.828223 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:48.828158    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 19:02:49.010631 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.010595    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-service-ca\") pod \"5d335e37-5496-4c3d-bc25-a292e7b5865a\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") "
Sep 16 19:02:49.010631 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.010635    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-config\") pod \"5d335e37-5496-4c3d-bc25-a292e7b5865a\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") "
Sep 16 19:02:49.010891 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.010674    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-oauth-serving-cert\") pod \"5d335e37-5496-4c3d-bc25-a292e7b5865a\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") "
Sep 16 19:02:49.010891 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.010694    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-trusted-ca-bundle\") pod \"5d335e37-5496-4c3d-bc25-a292e7b5865a\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") "
Sep 16 19:02:49.010891 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.010728    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d86zz\" (UniqueName: \"kubernetes.io/projected/5d335e37-5496-4c3d-bc25-a292e7b5865a-kube-api-access-d86zz\") pod \"5d335e37-5496-4c3d-bc25-a292e7b5865a\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") "
Sep 16 19:02:49.010891 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.010746    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-serving-cert\") pod \"5d335e37-5496-4c3d-bc25-a292e7b5865a\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") "
Sep 16 19:02:49.010891 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.010795    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-oauth-config\") pod \"5d335e37-5496-4c3d-bc25-a292e7b5865a\" (UID: \"5d335e37-5496-4c3d-bc25-a292e7b5865a\") "
Sep 16 19:02:49.011147 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.011077    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-service-ca" (OuterVolumeSpecName: "service-ca") pod "5d335e37-5496-4c3d-bc25-a292e7b5865a" (UID: "5d335e37-5496-4c3d-bc25-a292e7b5865a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:49.011147 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.011103    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5d335e37-5496-4c3d-bc25-a292e7b5865a" (UID: "5d335e37-5496-4c3d-bc25-a292e7b5865a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:49.011147 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.011125    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5d335e37-5496-4c3d-bc25-a292e7b5865a" (UID: "5d335e37-5496-4c3d-bc25-a292e7b5865a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:49.011483 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.011456    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-config" (OuterVolumeSpecName: "console-config") pod "5d335e37-5496-4c3d-bc25-a292e7b5865a" (UID: "5d335e37-5496-4c3d-bc25-a292e7b5865a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:49.013221 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.013193    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5d335e37-5496-4c3d-bc25-a292e7b5865a" (UID: "5d335e37-5496-4c3d-bc25-a292e7b5865a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:49.013221 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.013213    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5d335e37-5496-4c3d-bc25-a292e7b5865a" (UID: "5d335e37-5496-4c3d-bc25-a292e7b5865a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:49.013385 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.013310    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d335e37-5496-4c3d-bc25-a292e7b5865a-kube-api-access-d86zz" (OuterVolumeSpecName: "kube-api-access-d86zz") pod "5d335e37-5496-4c3d-bc25-a292e7b5865a" (UID: "5d335e37-5496-4c3d-bc25-a292e7b5865a"). InnerVolumeSpecName "kube-api-access-d86zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:49.111539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.111456    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86zz\" (UniqueName: \"kubernetes.io/projected/5d335e37-5496-4c3d-bc25-a292e7b5865a-kube-api-access-d86zz\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:02:49.111539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.111484    2584 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-serving-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:02:49.111539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.111494    2584 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-oauth-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:02:49.111539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.111504    2584 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-service-ca\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:02:49.111539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.111518    2584 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-console-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:02:49.111539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.111528    2584 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-oauth-serving-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:02:49.111539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.111536    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d335e37-5496-4c3d-bc25-a292e7b5865a-trusted-ca-bundle\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:02:49.578245 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.578207    2584 generic.go:356] "Generic (PLEG): container finished" podID="5d335e37-5496-4c3d-bc25-a292e7b5865a" containerID="63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b" exitCode=0
Sep 16 19:02:49.578656 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.578258    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b9bf4847-j9822"
Sep 16 19:02:49.578656 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.578289    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b9bf4847-j9822" event={"ID":"5d335e37-5496-4c3d-bc25-a292e7b5865a","Type":"ContainerDied","Data":"63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b"}
Sep 16 19:02:49.578656 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.578368    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b9bf4847-j9822" event={"ID":"5d335e37-5496-4c3d-bc25-a292e7b5865a","Type":"ContainerDied","Data":"8daa14cc967c028a0ab01eacc2cb9e7f17254b2818c1acfc3e1cc521e1e488e5"}
Sep 16 19:02:49.578656 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.578392    2584 scope.go:117] "RemoveContainer" containerID="63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b"
Sep 16 19:02:49.586988 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.586972    2584 scope.go:117] "RemoveContainer" containerID="63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b"
Sep 16 19:02:49.587239 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:02:49.587222    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b\": container with ID starting with 63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b not found: ID does not exist" containerID="63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b"
Sep 16 19:02:49.587364 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.587246    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b"} err="failed to get container status \"63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b\": rpc error: code = NotFound desc = could not find container \"63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b\": container with ID starting with 63e965faf6c69819859380e7659c0e26c990511226f1ddd135f90355ebdbef7b not found: ID does not exist"
Sep 16 19:02:49.606676 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.606649    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b9bf4847-j9822"]
Sep 16 19:02:49.614472 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.612438    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59b9bf4847-j9822"]
Sep 16 19:02:49.816776 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:02:49.816742    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d335e37-5496-4c3d-bc25-a292e7b5865a" path="/var/lib/kubelet/pods/5d335e37-5496-4c3d-bc25-a292e7b5865a/volumes"
Sep 16 19:03:27.266721 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.266685    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"]
Sep 16 19:03:27.267174 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.267046    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d335e37-5496-4c3d-bc25-a292e7b5865a" containerName="console"
Sep 16 19:03:27.267174 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.267062    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d335e37-5496-4c3d-bc25-a292e7b5865a" containerName="console"
Sep 16 19:03:27.267174 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.267138    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d335e37-5496-4c3d-bc25-a292e7b5865a" containerName="console"
Sep 16 19:03:27.270542 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.270518    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.273401 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.273373    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:27.273520 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.273424    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:27.273520 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.273435    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:27.273520 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.273378    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:03:27.274474 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.274456    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:03:27.274581 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.274481    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 16 19:03:27.274581 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.274522    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:27.282187 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.282163    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"]
Sep 16 19:03:27.304656 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.304628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhpw9\" (UniqueName: \"kubernetes.io/projected/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-api-access-lhpw9\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.304812 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.304667    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.304812 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.304702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.304812 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.304757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.304812 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.304778    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.405589 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.405554    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhpw9\" (UniqueName: \"kubernetes.io/projected/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-api-access-lhpw9\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.405589 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.405589    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.405846 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.405614    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.405846 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.405661    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.405846 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.405686    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.405846 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:03:27.405729    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:03:27.405846 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:03:27.405794    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-proxy-tls podName:74507e73-ca4d-4bf0-bfa3-66288e2e1df3 nodeName:}" failed. No retries permitted until 2026-09-16 19:03:27.905774441 +0000 UTC m=+1138.572713590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" (UID: "74507e73-ca4d-4bf0-bfa3-66288e2e1df3") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:03:27.406171 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.406149    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.406439 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.406422    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.406494 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.406423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.414863 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.414832    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhpw9\" (UniqueName: \"kubernetes.io/projected/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-api-access-lhpw9\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.911348 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.911306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:27.913715 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:27.913687    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:28.182570 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:28.182481    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:28.457653 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:28.457624    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"]
Sep 16 19:03:28.460184 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:03:28.460159    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74507e73_ca4d_4bf0_bfa3_66288e2e1df3.slice/crio-078bfeccff9be00397ca0f9a38214caee87eae325d1d017211d3571c57a6aa2d WatchSource:0}: Error finding container 078bfeccff9be00397ca0f9a38214caee87eae325d1d017211d3571c57a6aa2d: Status 404 returned error can't find the container with id 078bfeccff9be00397ca0f9a38214caee87eae325d1d017211d3571c57a6aa2d
Sep 16 19:03:28.462115 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:28.462098    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:03:28.753448 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:28.753364    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" event={"ID":"74507e73-ca4d-4bf0-bfa3-66288e2e1df3","Type":"ContainerStarted","Data":"078bfeccff9be00397ca0f9a38214caee87eae325d1d017211d3571c57a6aa2d"}
Sep 16 19:03:33.303638 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:33.303613    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:33.777863 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:33.777824    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" event={"ID":"74507e73-ca4d-4bf0-bfa3-66288e2e1df3","Type":"ContainerStarted","Data":"7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e"}
Sep 16 19:03:37.804015 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:37.803982    2584 generic.go:356] "Generic (PLEG): container finished" podID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerID="7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e" exitCode=0
Sep 16 19:03:37.804391 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:37.804067    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" event={"ID":"74507e73-ca4d-4bf0-bfa3-66288e2e1df3","Type":"ContainerDied","Data":"7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e"}
Sep 16 19:03:59.941257 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:59.941223    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" event={"ID":"74507e73-ca4d-4bf0-bfa3-66288e2e1df3","Type":"ContainerStarted","Data":"885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a"}
Sep 16 19:03:59.941668 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:59.941298    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" event={"ID":"74507e73-ca4d-4bf0-bfa3-66288e2e1df3","Type":"ContainerStarted","Data":"9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097"}
Sep 16 19:03:59.941668 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:59.941478    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:03:59.965513 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:03:59.965463    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" podStartSLOduration=2.742094068 podStartE2EDuration="32.965451138s" totalImagesPullingTime="26.426232192s" totalInitContainerRuntime="3.797124878s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:59.963930524 +0000 UTC m=+1170.630869704" watchObservedRunningTime="2026-09-16 19:03:59.965451138 +0000 UTC m=+1170.632390283"
Sep 16 19:04:00.946117 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:04:00.946086    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:04:06.956146 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:04:06.956116    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:04:36.957368 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:04:36.957320    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 19:04:46.957571 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:04:46.957529    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 19:04:56.957573 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:04:56.957530    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 19:05:06.957054 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:06.957009    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.36:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 16 19:05:17.009878 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:17.009839    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:05:27.354924 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:27.354836    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"]
Sep 16 19:05:27.355366 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:27.355073    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kserve-container" containerID="cri-o://9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097" gracePeriod=30
Sep 16 19:05:27.355366 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:27.355166    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kube-rbac-proxy" containerID="cri-o://885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a" gracePeriod=30
Sep 16 19:05:28.354074 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:28.354040    2584 generic.go:356] "Generic (PLEG): container finished" podID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerID="885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a" exitCode=2
Sep 16 19:05:28.354245 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:28.354099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" event={"ID":"74507e73-ca4d-4bf0-bfa3-66288e2e1df3","Type":"ContainerDied","Data":"885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a"}
Sep 16 19:05:31.950720 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:31.950677    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 16 19:05:32.208989 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.208919    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:05:32.285053 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.285015    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-proxy-tls\") pod \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") "
Sep 16 19:05:32.285226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.285063    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-rbac-proxy-sar-config\") pod \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") "
Sep 16 19:05:32.285226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.285096    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-cabundle-cert\") pod \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") "
Sep 16 19:05:32.285226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.285163    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kserve-provision-location\") pod \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") "
Sep 16 19:05:32.285421 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.285234    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhpw9\" (UniqueName: \"kubernetes.io/projected/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-api-access-lhpw9\") pod \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\" (UID: \"74507e73-ca4d-4bf0-bfa3-66288e2e1df3\") "
Sep 16 19:05:32.285579 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.285542    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74507e73-ca4d-4bf0-bfa3-66288e2e1df3" (UID: "74507e73-ca4d-4bf0-bfa3-66288e2e1df3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:32.285687 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.285549    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74507e73-ca4d-4bf0-bfa3-66288e2e1df3" (UID: "74507e73-ca4d-4bf0-bfa3-66288e2e1df3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:32.285687 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.285627    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74507e73-ca4d-4bf0-bfa3-66288e2e1df3" (UID: "74507e73-ca4d-4bf0-bfa3-66288e2e1df3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:32.287240 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.287220    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74507e73-ca4d-4bf0-bfa3-66288e2e1df3" (UID: "74507e73-ca4d-4bf0-bfa3-66288e2e1df3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:32.287358 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.287283    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-api-access-lhpw9" (OuterVolumeSpecName: "kube-api-access-lhpw9") pod "74507e73-ca4d-4bf0-bfa3-66288e2e1df3" (UID: "74507e73-ca4d-4bf0-bfa3-66288e2e1df3"). InnerVolumeSpecName "kube-api-access-lhpw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:32.375183 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.375142    2584 generic.go:356] "Generic (PLEG): container finished" podID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerID="9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097" exitCode=0
Sep 16 19:05:32.375374 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.375196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" event={"ID":"74507e73-ca4d-4bf0-bfa3-66288e2e1df3","Type":"ContainerDied","Data":"9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097"}
Sep 16 19:05:32.375374 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.375210    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"
Sep 16 19:05:32.375374 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.375309    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h" event={"ID":"74507e73-ca4d-4bf0-bfa3-66288e2e1df3","Type":"ContainerDied","Data":"078bfeccff9be00397ca0f9a38214caee87eae325d1d017211d3571c57a6aa2d"}
Sep 16 19:05:32.375374 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.375350    2584 scope.go:117] "RemoveContainer" containerID="885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a"
Sep 16 19:05:32.384744 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.384722    2584 scope.go:117] "RemoveContainer" containerID="9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097"
Sep 16 19:05:32.385925 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.385905    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.385993 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.385929    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.385993 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.385939    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.385993 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.385949    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.385993 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.385957    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhpw9\" (UniqueName: \"kubernetes.io/projected/74507e73-ca4d-4bf0-bfa3-66288e2e1df3-kube-api-access-lhpw9\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:05:32.393232 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.393214    2584 scope.go:117] "RemoveContainer" containerID="7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e"
Sep 16 19:05:32.400907 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.400853    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"]
Sep 16 19:05:32.402334 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.402311    2584 scope.go:117] "RemoveContainer" containerID="885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a"
Sep 16 19:05:32.402720 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:05:32.402700    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a\": container with ID starting with 885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a not found: ID does not exist" containerID="885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a"
Sep 16 19:05:32.402794 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.402729    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a"} err="failed to get container status \"885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a\": rpc error: code = NotFound desc = could not find container \"885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a\": container with ID starting with 885e7972e7e1945f26737d5c036549c4cd4797a13fddd7d1770920e9b6769e1a not found: ID does not exist"
Sep 16 19:05:32.402794 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.402749    2584 scope.go:117] "RemoveContainer" containerID="9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097"
Sep 16 19:05:32.403086 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:05:32.403046    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097\": container with ID starting with 9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097 not found: ID does not exist" containerID="9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097"
Sep 16 19:05:32.403195 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.403086    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097"} err="failed to get container status \"9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097\": rpc error: code = NotFound desc = could not find container \"9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097\": container with ID starting with 9446f624053bddd30caa2b2ffa5265ab30256de146b2a78d2b1ef5b8f1072097 not found: ID does not exist"
Sep 16 19:05:32.403195 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.403104    2584 scope.go:117] "RemoveContainer" containerID="7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e"
Sep 16 19:05:32.403439 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:05:32.403416    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e\": container with ID starting with 7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e not found: ID does not exist" containerID="7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e"
Sep 16 19:05:32.403503 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.403455    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e"} err="failed to get container status \"7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e\": rpc error: code = NotFound desc = could not find container \"7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e\": container with ID starting with 7a8761bf7a03cb7f9ea13f894547d68f6b4970a1e19b971b27a83534fa3ce65e not found: ID does not exist"
Sep 16 19:05:32.404135 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.404114    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-b452h"]
Sep 16 19:05:32.580215 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.580114    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"]
Sep 16 19:05:32.580714 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.580688    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kserve-container"
Sep 16 19:05:32.580714 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.580711    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kserve-container"
Sep 16 19:05:32.580873 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.580742    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="storage-initializer"
Sep 16 19:05:32.580873 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.580752    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="storage-initializer"
Sep 16 19:05:32.580873 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.580775    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kube-rbac-proxy"
Sep 16 19:05:32.580873 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.580785    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kube-rbac-proxy"
Sep 16 19:05:32.580873 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.580858    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kube-rbac-proxy"
Sep 16 19:05:32.580873 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.580870    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" containerName="kserve-container"
Sep 16 19:05:32.586170 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.586145    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.590043 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.590007    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:05:32.590181 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.590007    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:05:32.590181 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.590055    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 16 19:05:32.590181 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.590026    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:05:32.590181 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.590007    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:05:32.590419 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.590404    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:05:32.590655 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.590640    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:05:32.595943 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.595921    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"]
Sep 16 19:05:32.688748 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.688706    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.688748 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.688756    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd262\" (UniqueName: \"kubernetes.io/projected/9801c334-9306-40ef-866d-7d66858876db-kube-api-access-jd262\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.688979 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.688785    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9801c334-9306-40ef-866d-7d66858876db-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.688979 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.688815    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9801c334-9306-40ef-866d-7d66858876db-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.688979 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.688845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.790053 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.790012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.790309 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.790074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd262\" (UniqueName: \"kubernetes.io/projected/9801c334-9306-40ef-866d-7d66858876db-kube-api-access-jd262\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.790309 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.790105    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9801c334-9306-40ef-866d-7d66858876db-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.790309 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.790131    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9801c334-9306-40ef-866d-7d66858876db-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.790309 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.790166    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.790309 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:05:32.790292    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 16 19:05:32.790716 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:05:32.790370    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9801c334-9306-40ef-866d-7d66858876db-proxy-tls podName:9801c334-9306-40ef-866d-7d66858876db nodeName:}" failed. No retries permitted until 2026-09-16 19:05:33.290349227 +0000 UTC m=+1263.957288366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9801c334-9306-40ef-866d-7d66858876db-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" (UID: "9801c334-9306-40ef-866d-7d66858876db") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 16 19:05:32.790716 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.790651    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9801c334-9306-40ef-866d-7d66858876db-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.790990 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.790966    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.791058 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.791044    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:32.801479 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:32.801452    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd262\" (UniqueName: \"kubernetes.io/projected/9801c334-9306-40ef-866d-7d66858876db-kube-api-access-jd262\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:33.294313 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:33.294245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9801c334-9306-40ef-866d-7d66858876db-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:33.296713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:33.296692    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9801c334-9306-40ef-866d-7d66858876db-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:33.498191 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:33.498150    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:33.774421 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:33.774395    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"]
Sep 16 19:05:33.776968 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:05:33.776943    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9801c334_9306_40ef_866d_7d66858876db.slice/crio-208a58524222e6262940235cbb488eaf7f689263bc3352124de3edabcbfeb189 WatchSource:0}: Error finding container 208a58524222e6262940235cbb488eaf7f689263bc3352124de3edabcbfeb189: Status 404 returned error can't find the container with id 208a58524222e6262940235cbb488eaf7f689263bc3352124de3edabcbfeb189
Sep 16 19:05:33.817090 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:33.817056    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74507e73-ca4d-4bf0-bfa3-66288e2e1df3" path="/var/lib/kubelet/pods/74507e73-ca4d-4bf0-bfa3-66288e2e1df3/volumes"
Sep 16 19:05:34.386186 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:34.386150    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" event={"ID":"9801c334-9306-40ef-866d-7d66858876db","Type":"ContainerStarted","Data":"7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3"}
Sep 16 19:05:34.386645 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:34.386216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" event={"ID":"9801c334-9306-40ef-866d-7d66858876db","Type":"ContainerStarted","Data":"208a58524222e6262940235cbb488eaf7f689263bc3352124de3edabcbfeb189"}
Sep 16 19:05:38.404869 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:38.404836    2584 generic.go:356] "Generic (PLEG): container finished" podID="9801c334-9306-40ef-866d-7d66858876db" containerID="7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3" exitCode=0
Sep 16 19:05:38.405305 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:38.404908    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" event={"ID":"9801c334-9306-40ef-866d-7d66858876db","Type":"ContainerDied","Data":"7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3"}
Sep 16 19:05:39.412166 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:39.412125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" event={"ID":"9801c334-9306-40ef-866d-7d66858876db","Type":"ContainerStarted","Data":"4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885"}
Sep 16 19:05:39.412578 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:39.412202    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" event={"ID":"9801c334-9306-40ef-866d-7d66858876db","Type":"ContainerStarted","Data":"f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71"}
Sep 16 19:05:39.412578 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:39.412444    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:39.434642 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:39.434583    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" podStartSLOduration=3.613362497 podStartE2EDuration="7.434566846s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.821204349s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:39.432914079 +0000 UTC m=+1270.099853226" watchObservedRunningTime="2026-09-16 19:05:39.434566846 +0000 UTC m=+1270.101505991"
Sep 16 19:05:40.417333 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:40.417299    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:05:46.427930 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:05:46.427893    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:06:16.428975 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:06:16.428933    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:06:26.429330 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:06:26.429289    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:06:36.429251 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:06:36.429202    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:06:46.428729 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:06:46.428688    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:06:49.813426 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:06:49.813384    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 16 19:06:59.910715 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:06:59.910669    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:07:02.627098 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:02.627065    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"]
Sep 16 19:07:02.627559 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:02.627373    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kserve-container" containerID="cri-o://f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71" gracePeriod=30
Sep 16 19:07:02.627559 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:02.627500    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kube-rbac-proxy" containerID="cri-o://4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885" gracePeriod=30
Sep 16 19:07:02.793348 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:02.793317    2584 generic.go:356] "Generic (PLEG): container finished" podID="9801c334-9306-40ef-866d-7d66858876db" containerID="4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885" exitCode=2
Sep 16 19:07:02.793513 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:02.793391    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" event={"ID":"9801c334-9306-40ef-866d-7d66858876db","Type":"ContainerDied","Data":"4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885"}
Sep 16 19:07:06.423237 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:06.423195    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 16 19:07:07.572415 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.572389    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:07:07.707300 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.707228    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9801c334-9306-40ef-866d-7d66858876db-kserve-provision-location\") pod \"9801c334-9306-40ef-866d-7d66858876db\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") "
Sep 16 19:07:07.707549 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.707356    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-cabundle-cert\") pod \"9801c334-9306-40ef-866d-7d66858876db\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") "
Sep 16 19:07:07.707549 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.707397    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-kube-rbac-proxy-sar-config\") pod \"9801c334-9306-40ef-866d-7d66858876db\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") "
Sep 16 19:07:07.707549 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.707448    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9801c334-9306-40ef-866d-7d66858876db-proxy-tls\") pod \"9801c334-9306-40ef-866d-7d66858876db\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") "
Sep 16 19:07:07.707549 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.707497    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd262\" (UniqueName: \"kubernetes.io/projected/9801c334-9306-40ef-866d-7d66858876db-kube-api-access-jd262\") pod \"9801c334-9306-40ef-866d-7d66858876db\" (UID: \"9801c334-9306-40ef-866d-7d66858876db\") "
Sep 16 19:07:07.707836 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.707656    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9801c334-9306-40ef-866d-7d66858876db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9801c334-9306-40ef-866d-7d66858876db" (UID: "9801c334-9306-40ef-866d-7d66858876db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:07.707836 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.707773    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9801c334-9306-40ef-866d-7d66858876db-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:07:07.708120 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.708084    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9801c334-9306-40ef-866d-7d66858876db" (UID: "9801c334-9306-40ef-866d-7d66858876db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:07.708548 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.708526    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9801c334-9306-40ef-866d-7d66858876db" (UID: "9801c334-9306-40ef-866d-7d66858876db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:07.709951 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.709923    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9801c334-9306-40ef-866d-7d66858876db-kube-api-access-jd262" (OuterVolumeSpecName: "kube-api-access-jd262") pod "9801c334-9306-40ef-866d-7d66858876db" (UID: "9801c334-9306-40ef-866d-7d66858876db"). InnerVolumeSpecName "kube-api-access-jd262". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:07.710038 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.710016    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9801c334-9306-40ef-866d-7d66858876db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9801c334-9306-40ef-866d-7d66858876db" (UID: "9801c334-9306-40ef-866d-7d66858876db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:07.808442 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.808397    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9801c334-9306-40ef-866d-7d66858876db-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:07:07.808442 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.808432    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd262\" (UniqueName: \"kubernetes.io/projected/9801c334-9306-40ef-866d-7d66858876db-kube-api-access-jd262\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:07:07.808442 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.808444    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:07:07.808442 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.808453    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9801c334-9306-40ef-866d-7d66858876db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:07:07.820599 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.820562    2584 generic.go:356] "Generic (PLEG): container finished" podID="9801c334-9306-40ef-866d-7d66858876db" containerID="f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71" exitCode=0
Sep 16 19:07:07.820811 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.820630    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"
Sep 16 19:07:07.820811 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.820630    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" event={"ID":"9801c334-9306-40ef-866d-7d66858876db","Type":"ContainerDied","Data":"f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71"}
Sep 16 19:07:07.820811 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.820774    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm" event={"ID":"9801c334-9306-40ef-866d-7d66858876db","Type":"ContainerDied","Data":"208a58524222e6262940235cbb488eaf7f689263bc3352124de3edabcbfeb189"}
Sep 16 19:07:07.820990 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.820815    2584 scope.go:117] "RemoveContainer" containerID="4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885"
Sep 16 19:07:07.830851 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.830828    2584 scope.go:117] "RemoveContainer" containerID="f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71"
Sep 16 19:07:07.838806 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.838787    2584 scope.go:117] "RemoveContainer" containerID="7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3"
Sep 16 19:07:07.847027 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.846996    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"]
Sep 16 19:07:07.847235 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.847218    2584 scope.go:117] "RemoveContainer" containerID="4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885"
Sep 16 19:07:07.847577 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:07:07.847558    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885\": container with ID starting with 4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885 not found: ID does not exist" containerID="4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885"
Sep 16 19:07:07.847632 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.847588    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885"} err="failed to get container status \"4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885\": rpc error: code = NotFound desc = could not find container \"4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885\": container with ID starting with 4db7dd95795747ec47bec703896be9cd95f953c901a6847dc2142850c7323885 not found: ID does not exist"
Sep 16 19:07:07.847632 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.847610    2584 scope.go:117] "RemoveContainer" containerID="f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71"
Sep 16 19:07:07.847877 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:07:07.847860    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71\": container with ID starting with f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71 not found: ID does not exist" containerID="f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71"
Sep 16 19:07:07.847920 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.847880    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71"} err="failed to get container status \"f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71\": rpc error: code = NotFound desc = could not find container \"f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71\": container with ID starting with f564dd81fcbbb532a32097c0ec18c9559f223b585b2650fdae525fd226c2de71 not found: ID does not exist"
Sep 16 19:07:07.847920 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.847893    2584 scope.go:117] "RemoveContainer" containerID="7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3"
Sep 16 19:07:07.848144 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:07:07.848125    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3\": container with ID starting with 7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3 not found: ID does not exist" containerID="7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3"
Sep 16 19:07:07.848184 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.848152    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3"} err="failed to get container status \"7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3\": rpc error: code = NotFound desc = could not find container \"7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3\": container with ID starting with 7157352ccd7fb1ca76697693c587497f369f69852d4d36249fd4fb94a4363ca3 not found: ID does not exist"
Sep 16 19:07:07.854700 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:07.854673    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-5x6wm"]
Sep 16 19:07:09.817375 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:09.817343    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9801c334-9306-40ef-866d-7d66858876db" path="/var/lib/kubelet/pods/9801c334-9306-40ef-866d-7d66858876db/volumes"
Sep 16 19:07:25.380144 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.380106    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-5f6e9bf4"]
Sep 16 19:07:25.380633 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.380479    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="storage-initializer"
Sep 16 19:07:25.380633 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.380495    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="storage-initializer"
Sep 16 19:07:25.380633 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.380514    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kube-rbac-proxy"
Sep 16 19:07:25.380633 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.380537    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kube-rbac-proxy"
Sep 16 19:07:25.380633 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.380566    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kserve-container"
Sep 16 19:07:25.380633 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.380573    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kserve-container"
Sep 16 19:07:25.380633 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.380620    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kube-rbac-proxy"
Sep 16 19:07:25.380633 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.380630    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9801c334-9306-40ef-866d-7d66858876db" containerName="kserve-container"
Sep 16 19:07:25.385206 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.385188    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-5f6e9bf4"
Sep 16 19:07:25.389635 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.389607    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-5f6e9bf4"]
Sep 16 19:07:25.561311 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.561260    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5dr\" (UniqueName: \"kubernetes.io/projected/116e4ca1-e813-4067-a9bb-d4d0a87f8be3-kube-api-access-dp5dr\") pod \"curl-metrics-5f6e9bf4\" (UID: \"116e4ca1-e813-4067-a9bb-d4d0a87f8be3\") " pod="kserve/curl-metrics-5f6e9bf4"
Sep 16 19:07:25.661997 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.661963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5dr\" (UniqueName: \"kubernetes.io/projected/116e4ca1-e813-4067-a9bb-d4d0a87f8be3-kube-api-access-dp5dr\") pod \"curl-metrics-5f6e9bf4\" (UID: \"116e4ca1-e813-4067-a9bb-d4d0a87f8be3\") " pod="kserve/curl-metrics-5f6e9bf4"
Sep 16 19:07:25.671539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.671515    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5dr\" (UniqueName: \"kubernetes.io/projected/116e4ca1-e813-4067-a9bb-d4d0a87f8be3-kube-api-access-dp5dr\") pod \"curl-metrics-5f6e9bf4\" (UID: \"116e4ca1-e813-4067-a9bb-d4d0a87f8be3\") " pod="kserve/curl-metrics-5f6e9bf4"
Sep 16 19:07:25.702376 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.702345    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-5f6e9bf4"
Sep 16 19:07:25.970598 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:25.970523    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-5f6e9bf4"]
Sep 16 19:07:25.972890 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:07:25.972854    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod116e4ca1_e813_4067_a9bb_d4d0a87f8be3.slice/crio-0760b3fdf7bad62a212dfe543c53c823e8405f52f485d03f838e0a4cbc72ded3 WatchSource:0}: Error finding container 0760b3fdf7bad62a212dfe543c53c823e8405f52f485d03f838e0a4cbc72ded3: Status 404 returned error can't find the container with id 0760b3fdf7bad62a212dfe543c53c823e8405f52f485d03f838e0a4cbc72ded3
Sep 16 19:07:26.911037 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:26.911000    2584 generic.go:356] "Generic (PLEG): container finished" podID="116e4ca1-e813-4067-a9bb-d4d0a87f8be3" containerID="8f9d0b8ba3b91bf8e877dd8f541c28585a77d0c745aee4428fcbd6906658d472" exitCode=0
Sep 16 19:07:26.911487 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:26.911063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-5f6e9bf4" event={"ID":"116e4ca1-e813-4067-a9bb-d4d0a87f8be3","Type":"ContainerDied","Data":"8f9d0b8ba3b91bf8e877dd8f541c28585a77d0c745aee4428fcbd6906658d472"}
Sep 16 19:07:26.911487 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:26.911093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-5f6e9bf4" event={"ID":"116e4ca1-e813-4067-a9bb-d4d0a87f8be3","Type":"ContainerStarted","Data":"0760b3fdf7bad62a212dfe543c53c823e8405f52f485d03f838e0a4cbc72ded3"}
Sep 16 19:07:28.043065 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:28.043034    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-5f6e9bf4"
Sep 16 19:07:28.082076 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:28.082049    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-5f6e9bf4_116e4ca1-e813-4067-a9bb-d4d0a87f8be3/curl-metrics-5f6e9bf4/0.log"
Sep 16 19:07:28.083242 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:28.083222    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp5dr\" (UniqueName: \"kubernetes.io/projected/116e4ca1-e813-4067-a9bb-d4d0a87f8be3-kube-api-access-dp5dr\") pod \"116e4ca1-e813-4067-a9bb-d4d0a87f8be3\" (UID: \"116e4ca1-e813-4067-a9bb-d4d0a87f8be3\") "
Sep 16 19:07:28.085170 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:28.085149    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116e4ca1-e813-4067-a9bb-d4d0a87f8be3-kube-api-access-dp5dr" (OuterVolumeSpecName: "kube-api-access-dp5dr") pod "116e4ca1-e813-4067-a9bb-d4d0a87f8be3" (UID: "116e4ca1-e813-4067-a9bb-d4d0a87f8be3"). InnerVolumeSpecName "kube-api-access-dp5dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:28.107114 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:28.107084    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-5f6e9bf4"]
Sep 16 19:07:28.109413 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:28.109389    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-5f6e9bf4"]
Sep 16 19:07:28.184686 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:28.184574    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp5dr\" (UniqueName: \"kubernetes.io/projected/116e4ca1-e813-4067-a9bb-d4d0a87f8be3-kube-api-access-dp5dr\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:07:28.921860 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:28.921824    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-5f6e9bf4"
Sep 16 19:07:28.922028 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:28.921831    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0760b3fdf7bad62a212dfe543c53c823e8405f52f485d03f838e0a4cbc72ded3"
Sep 16 19:07:28.932214 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:07:28.932181    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-5f6e9bf4\" is forbidden: User \"system:node:ip-10-0-139-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-139-120.ec2.internal' and this object" podUID="116e4ca1-e813-4067-a9bb-d4d0a87f8be3" pod="kserve/curl-metrics-5f6e9bf4"
Sep 16 19:07:29.815383 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:07:29.815338    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-5f6e9bf4\" is forbidden: User \"system:node:ip-10-0-139-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-139-120.ec2.internal' and this object" podUID="116e4ca1-e813-4067-a9bb-d4d0a87f8be3" pod="kserve/curl-metrics-5f6e9bf4"
Sep 16 19:07:29.816441 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:07:29.816418    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="116e4ca1-e813-4067-a9bb-d4d0a87f8be3" path="/var/lib/kubelet/pods/116e4ca1-e813-4067-a9bb-d4d0a87f8be3/volumes"
Sep 16 19:08:40.737986 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.737947    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"]
Sep 16 19:08:40.738550 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.738504    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116e4ca1-e813-4067-a9bb-d4d0a87f8be3" containerName="curl-metrics-5f6e9bf4"
Sep 16 19:08:40.738550 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.738525    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="116e4ca1-e813-4067-a9bb-d4d0a87f8be3" containerName="curl-metrics-5f6e9bf4"
Sep 16 19:08:40.738675 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.738601    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="116e4ca1-e813-4067-a9bb-d4d0a87f8be3" containerName="curl-metrics-5f6e9bf4"
Sep 16 19:08:40.741992 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.741969    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.744874 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.744849    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:08:40.744874 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.744865    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:08:40.745034 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.744865    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:08:40.745034 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.745003    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 16 19:08:40.745256 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.745232    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:08:40.745341 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.745327    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:08:40.745962 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.745948    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:08:40.755813 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.755792    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"]
Sep 16 19:08:40.768331 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.768304    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prv8d\" (UniqueName: \"kubernetes.io/projected/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-api-access-prv8d\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.768425 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.768395    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.768466 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.768433    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ca6d67-c885-4ecf-b48c-8bef21902974-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.768501 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.768468    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.768501 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.768492    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ca6d67-c885-4ecf-b48c-8bef21902974-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.869330 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.869306    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ca6d67-c885-4ecf-b48c-8bef21902974-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.869471 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.869357    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.869471 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.869394    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ca6d67-c885-4ecf-b48c-8bef21902974-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.869591 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.869484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prv8d\" (UniqueName: \"kubernetes.io/projected/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-api-access-prv8d\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.869669 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.869641    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.869772 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.869753    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ca6d67-c885-4ecf-b48c-8bef21902974-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.869999 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:08:40.869978    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 16 19:08:40.870067 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:08:40.870044    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ca6d67-c885-4ecf-b48c-8bef21902974-proxy-tls podName:a0ca6d67-c885-4ecf-b48c-8bef21902974 nodeName:}" failed. No retries permitted until 2026-09-16 19:08:41.370025225 +0000 UTC m=+1452.036964349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a0ca6d67-c885-4ecf-b48c-8bef21902974-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" (UID: "a0ca6d67-c885-4ecf-b48c-8bef21902974") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 16 19:08:40.870534 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.870513    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.870534 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.870530    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:40.881100 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:40.881072    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prv8d\" (UniqueName: \"kubernetes.io/projected/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-api-access-prv8d\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:41.373739 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:41.373710    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ca6d67-c885-4ecf-b48c-8bef21902974-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:41.376147 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:41.376124    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ca6d67-c885-4ecf-b48c-8bef21902974-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:41.652387 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:41.652294    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:08:41.940760 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:41.940658    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"]
Sep 16 19:08:41.943062 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:08:41.943034    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ca6d67_c885_4ecf_b48c_8bef21902974.slice/crio-529c1214816651d1654b64300f7e1f4676a2d2f841cf5e5ce860c5fa814a6d70 WatchSource:0}: Error finding container 529c1214816651d1654b64300f7e1f4676a2d2f841cf5e5ce860c5fa814a6d70: Status 404 returned error can't find the container with id 529c1214816651d1654b64300f7e1f4676a2d2f841cf5e5ce860c5fa814a6d70
Sep 16 19:08:42.240217 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:42.240136    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" event={"ID":"a0ca6d67-c885-4ecf-b48c-8bef21902974","Type":"ContainerStarted","Data":"a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a"}
Sep 16 19:08:42.240217 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:08:42.240209    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" event={"ID":"a0ca6d67-c885-4ecf-b48c-8bef21902974","Type":"ContainerStarted","Data":"529c1214816651d1654b64300f7e1f4676a2d2f841cf5e5ce860c5fa814a6d70"}
Sep 16 19:09:11.370086 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:09:11.370055    2584 generic.go:356] "Generic (PLEG): container finished" podID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerID="a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a" exitCode=0
Sep 16 19:09:11.370475 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:09:11.370114    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" event={"ID":"a0ca6d67-c885-4ecf-b48c-8bef21902974","Type":"ContainerDied","Data":"a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a"}
Sep 16 19:09:11.371479 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:09:11.371460    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:11:13.993402 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:11:13.993369    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" event={"ID":"a0ca6d67-c885-4ecf-b48c-8bef21902974","Type":"ContainerStarted","Data":"156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5"}
Sep 16 19:11:15.000551 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:11:15.000508    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" event={"ID":"a0ca6d67-c885-4ecf-b48c-8bef21902974","Type":"ContainerStarted","Data":"1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b"}
Sep 16 19:11:15.000979 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:11:15.000588    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:11:15.000979 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:11:15.000621    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:11:15.033136 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:11:15.033083    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" podStartSLOduration=3.32065556 podStartE2EDuration="2m35.033065899s" totalImagesPullingTime="2m2.372250567s" totalInitContainerRuntime="29.340159772s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:15.030613436 +0000 UTC m=+1605.697552637" watchObservedRunningTime="2026-09-16 19:11:15.033065899 +0000 UTC m=+1605.700005046"
Sep 16 19:11:21.013215 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:11:21.013182    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:12:52.814913 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:12:52.814820    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:13:03.852448 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:03.852400    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:13:06.993122 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:06.993091    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"]
Sep 16 19:13:06.993530 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:06.993354    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="kserve-container" containerID="cri-o://156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5" gracePeriod=30
Sep 16 19:13:06.993530 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:06.993407    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="kube-rbac-proxy" containerID="cri-o://1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b" gracePeriod=30
Sep 16 19:13:07.504465 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:07.504428    2584 generic.go:356] "Generic (PLEG): container finished" podID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerID="1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b" exitCode=2
Sep 16 19:13:07.504636 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:07.504492    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" event={"ID":"a0ca6d67-c885-4ecf-b48c-8bef21902974","Type":"ContainerDied","Data":"1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b"}
Sep 16 19:13:08.331246 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.331223    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:13:08.434418 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.434388    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-cabundle-cert\") pod \"a0ca6d67-c885-4ecf-b48c-8bef21902974\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") "
Sep 16 19:13:08.434575 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.434444    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ca6d67-c885-4ecf-b48c-8bef21902974-kserve-provision-location\") pod \"a0ca6d67-c885-4ecf-b48c-8bef21902974\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") "
Sep 16 19:13:08.434575 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.434468    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-rbac-proxy-sar-config\") pod \"a0ca6d67-c885-4ecf-b48c-8bef21902974\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") "
Sep 16 19:13:08.434575 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.434519    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ca6d67-c885-4ecf-b48c-8bef21902974-proxy-tls\") pod \"a0ca6d67-c885-4ecf-b48c-8bef21902974\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") "
Sep 16 19:13:08.434685 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.434659    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prv8d\" (UniqueName: \"kubernetes.io/projected/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-api-access-prv8d\") pod \"a0ca6d67-c885-4ecf-b48c-8bef21902974\" (UID: \"a0ca6d67-c885-4ecf-b48c-8bef21902974\") "
Sep 16 19:13:08.434857 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.434831    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a0ca6d67-c885-4ecf-b48c-8bef21902974" (UID: "a0ca6d67-c885-4ecf-b48c-8bef21902974"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:08.434926 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.434864    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0ca6d67-c885-4ecf-b48c-8bef21902974" (UID: "a0ca6d67-c885-4ecf-b48c-8bef21902974"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:08.434995 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.434978    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:08.435045 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.434998    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca6d67-c885-4ecf-b48c-8bef21902974-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:08.436656 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.436633    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ca6d67-c885-4ecf-b48c-8bef21902974-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0ca6d67-c885-4ecf-b48c-8bef21902974" (UID: "a0ca6d67-c885-4ecf-b48c-8bef21902974"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:08.436656 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.436634    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-api-access-prv8d" (OuterVolumeSpecName: "kube-api-access-prv8d") pod "a0ca6d67-c885-4ecf-b48c-8bef21902974" (UID: "a0ca6d67-c885-4ecf-b48c-8bef21902974"). InnerVolumeSpecName "kube-api-access-prv8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:08.438064 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.438043    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca6d67-c885-4ecf-b48c-8bef21902974-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0ca6d67-c885-4ecf-b48c-8bef21902974" (UID: "a0ca6d67-c885-4ecf-b48c-8bef21902974"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:08.511374 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.511344    2584 generic.go:356] "Generic (PLEG): container finished" podID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerID="156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5" exitCode=0
Sep 16 19:13:08.511538 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.511388    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" event={"ID":"a0ca6d67-c885-4ecf-b48c-8bef21902974","Type":"ContainerDied","Data":"156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5"}
Sep 16 19:13:08.511538 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.511407    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"
Sep 16 19:13:08.511538 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.511435    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn" event={"ID":"a0ca6d67-c885-4ecf-b48c-8bef21902974","Type":"ContainerDied","Data":"529c1214816651d1654b64300f7e1f4676a2d2f841cf5e5ce860c5fa814a6d70"}
Sep 16 19:13:08.511538 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.511462    2584 scope.go:117] "RemoveContainer" containerID="1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b"
Sep 16 19:13:08.519615 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.519497    2584 scope.go:117] "RemoveContainer" containerID="156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5"
Sep 16 19:13:08.526865 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.526848    2584 scope.go:117] "RemoveContainer" containerID="a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a"
Sep 16 19:13:08.534228 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.534214    2584 scope.go:117] "RemoveContainer" containerID="1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b"
Sep 16 19:13:08.534531 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:13:08.534512    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b\": container with ID starting with 1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b not found: ID does not exist" containerID="1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b"
Sep 16 19:13:08.534582 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.534539    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b"} err="failed to get container status \"1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b\": rpc error: code = NotFound desc = could not find container \"1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b\": container with ID starting with 1fd8d1cd27263cf3d311e4bf73e5fa5321700443fef6001a6ee65b096bf22a9b not found: ID does not exist"
Sep 16 19:13:08.534582 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.534557    2584 scope.go:117] "RemoveContainer" containerID="156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5"
Sep 16 19:13:08.534828 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:13:08.534813    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5\": container with ID starting with 156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5 not found: ID does not exist" containerID="156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5"
Sep 16 19:13:08.534881 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.534832    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5"} err="failed to get container status \"156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5\": rpc error: code = NotFound desc = could not find container \"156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5\": container with ID starting with 156d9cc22d156e2b722c6bbf4712cb8788fcb8e5c1177376e845c774463b31c5 not found: ID does not exist"
Sep 16 19:13:08.534881 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.534848    2584 scope.go:117] "RemoveContainer" containerID="a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a"
Sep 16 19:13:08.535108 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:13:08.535090    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a\": container with ID starting with a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a not found: ID does not exist" containerID="a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a"
Sep 16 19:13:08.535150 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.535114    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a"} err="failed to get container status \"a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a\": rpc error: code = NotFound desc = could not find container \"a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a\": container with ID starting with a076158819846eb1fa074f799443eba79baaddf5f75f40ccf637c7f8c1f1e39a not found: ID does not exist"
Sep 16 19:13:08.535392 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.535374    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0ca6d67-c885-4ecf-b48c-8bef21902974-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:08.535442 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.535395    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ca6d67-c885-4ecf-b48c-8bef21902974-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:08.535442 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.535405    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prv8d\" (UniqueName: \"kubernetes.io/projected/a0ca6d67-c885-4ecf-b48c-8bef21902974-kube-api-access-prv8d\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:08.538756 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.538734    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"]
Sep 16 19:13:08.543917 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:08.543896    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-kt2kn"]
Sep 16 19:13:09.816690 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:09.816653    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" path="/var/lib/kubelet/pods/a0ca6d67-c885-4ecf-b48c-8bef21902974/volumes"
Sep 16 19:13:12.194381 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.194347    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"]
Sep 16 19:13:12.194742 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.194692    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="kserve-container"
Sep 16 19:13:12.194742 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.194707    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="kserve-container"
Sep 16 19:13:12.194742 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.194730    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="kube-rbac-proxy"
Sep 16 19:13:12.194742 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.194735    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="kube-rbac-proxy"
Sep 16 19:13:12.194874 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.194744    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="storage-initializer"
Sep 16 19:13:12.194874 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.194750    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="storage-initializer"
Sep 16 19:13:12.194874 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.194802    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="kube-rbac-proxy"
Sep 16 19:13:12.194874 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.194812    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ca6d67-c885-4ecf-b48c-8bef21902974" containerName="kserve-container"
Sep 16 19:13:12.199862 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.199841    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.204012 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.203987    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:13:12.204132 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.204016    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:13:12.204335 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.204318    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:13:12.205141 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.205121    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 16 19:13:12.205253 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.205125    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:13:12.205253 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.205162    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 16 19:13:12.206959 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.206586    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:13:12.212667 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.212644    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"]
Sep 16 19:13:12.266824 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.266797    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3caed81-90b9-4446-8c57-aa7b24ac3784-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.266997 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.266835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.266997 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.266971    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.267118 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.267012    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9c7k\" (UniqueName: \"kubernetes.io/projected/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-api-access-f9c7k\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.267118 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.267063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3caed81-90b9-4446-8c57-aa7b24ac3784-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.368010 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.367972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3caed81-90b9-4446-8c57-aa7b24ac3784-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.368010 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.368013    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.368243 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.368074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.368243 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.368104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9c7k\" (UniqueName: \"kubernetes.io/projected/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-api-access-f9c7k\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.368243 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.368225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3caed81-90b9-4446-8c57-aa7b24ac3784-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.368607 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.368577    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3caed81-90b9-4446-8c57-aa7b24ac3784-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.368794 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.368774    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.368877 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.368830    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.370463 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.370444    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3caed81-90b9-4446-8c57-aa7b24ac3784-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.378155 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.378134    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9c7k\" (UniqueName: \"kubernetes.io/projected/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-api-access-f9c7k\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.511633 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.511555    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:12.791242 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:12.791217    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"]
Sep 16 19:13:12.793388 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:13:12.793363    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3caed81_90b9_4446_8c57_aa7b24ac3784.slice/crio-8dccd452b644bbdd86a04eaf5bc1ab9f7c2d5938526a5a3f6d9e5463583398a6 WatchSource:0}: Error finding container 8dccd452b644bbdd86a04eaf5bc1ab9f7c2d5938526a5a3f6d9e5463583398a6: Status 404 returned error can't find the container with id 8dccd452b644bbdd86a04eaf5bc1ab9f7c2d5938526a5a3f6d9e5463583398a6
Sep 16 19:13:13.535953 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:13.535913    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" event={"ID":"f3caed81-90b9-4446-8c57-aa7b24ac3784","Type":"ContainerStarted","Data":"d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0"}
Sep 16 19:13:13.536337 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:13.535985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" event={"ID":"f3caed81-90b9-4446-8c57-aa7b24ac3784","Type":"ContainerStarted","Data":"8dccd452b644bbdd86a04eaf5bc1ab9f7c2d5938526a5a3f6d9e5463583398a6"}
Sep 16 19:13:22.577184 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:22.577148    2584 generic.go:356] "Generic (PLEG): container finished" podID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerID="d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0" exitCode=0
Sep 16 19:13:22.577577 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:22.577197    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" event={"ID":"f3caed81-90b9-4446-8c57-aa7b24ac3784","Type":"ContainerDied","Data":"d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0"}
Sep 16 19:13:23.585224 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:23.585187    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" event={"ID":"f3caed81-90b9-4446-8c57-aa7b24ac3784","Type":"ContainerStarted","Data":"2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9"}
Sep 16 19:13:23.585595 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:23.585285    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" event={"ID":"f3caed81-90b9-4446-8c57-aa7b24ac3784","Type":"ContainerStarted","Data":"b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf"}
Sep 16 19:13:23.585595 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:23.585502    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:23.610884 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:23.610838    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" podStartSLOduration=2.700817226 podStartE2EDuration="11.610824071s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.910006845s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:23.609451634 +0000 UTC m=+1734.276390806" watchObservedRunningTime="2026-09-16 19:13:23.610824071 +0000 UTC m=+1734.277763216"
Sep 16 19:13:24.589485 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:24.589450    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:24.590832 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:24.590806    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 16 19:13:25.594258 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:25.594214    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 16 19:13:30.471462 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:30.471431    2584 scope.go:117] "RemoveContainer" containerID="8f9d0b8ba3b91bf8e877dd8f541c28585a77d0c745aee4428fcbd6906658d472"
Sep 16 19:13:30.599189 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:30.599161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:30.599782 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:30.599756    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 16 19:13:40.600972 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:40.600942    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:54.148196 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:54.148162    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"]
Sep 16 19:13:54.148677 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:54.148422    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kserve-container" containerID="cri-o://b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf" gracePeriod=30
Sep 16 19:13:54.148677 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:54.148481    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kube-rbac-proxy" containerID="cri-o://2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9" gracePeriod=30
Sep 16 19:13:54.733778 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:54.733744    2584 generic.go:356] "Generic (PLEG): container finished" podID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerID="2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9" exitCode=2
Sep 16 19:13:54.733953 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:54.733797    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" event={"ID":"f3caed81-90b9-4446-8c57-aa7b24ac3784","Type":"ContainerDied","Data":"2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9"}
Sep 16 19:13:55.594798 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.594756    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 16 19:13:55.697761 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.697736    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:55.740670 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.740602    2584 generic.go:356] "Generic (PLEG): container finished" podID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerID="b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf" exitCode=0
Sep 16 19:13:55.740670 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.740656    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"
Sep 16 19:13:55.740806 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.740656    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" event={"ID":"f3caed81-90b9-4446-8c57-aa7b24ac3784","Type":"ContainerDied","Data":"b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf"}
Sep 16 19:13:55.740840 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.740806    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt" event={"ID":"f3caed81-90b9-4446-8c57-aa7b24ac3784","Type":"ContainerDied","Data":"8dccd452b644bbdd86a04eaf5bc1ab9f7c2d5938526a5a3f6d9e5463583398a6"}
Sep 16 19:13:55.740873 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.740839    2584 scope.go:117] "RemoveContainer" containerID="2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9"
Sep 16 19:13:55.748816 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.748759    2584 scope.go:117] "RemoveContainer" containerID="b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf"
Sep 16 19:13:55.756004 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.755988    2584 scope.go:117] "RemoveContainer" containerID="d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0"
Sep 16 19:13:55.763116 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.763102    2584 scope.go:117] "RemoveContainer" containerID="2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9"
Sep 16 19:13:55.763440 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:13:55.763422    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9\": container with ID starting with 2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9 not found: ID does not exist" containerID="2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9"
Sep 16 19:13:55.763486 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.763448    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9"} err="failed to get container status \"2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9\": rpc error: code = NotFound desc = could not find container \"2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9\": container with ID starting with 2262821ab219a80faef303b90e299837e305449e0a531ae282487937deb038e9 not found: ID does not exist"
Sep 16 19:13:55.763486 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.763465    2584 scope.go:117] "RemoveContainer" containerID="b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf"
Sep 16 19:13:55.763705 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:13:55.763686    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf\": container with ID starting with b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf not found: ID does not exist" containerID="b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf"
Sep 16 19:13:55.763746 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.763714    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf"} err="failed to get container status \"b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf\": rpc error: code = NotFound desc = could not find container \"b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf\": container with ID starting with b83af5351df4c6e061d78e755ee400e502ffd7dec3f0eedd78770adb0e42afcf not found: ID does not exist"
Sep 16 19:13:55.763746 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.763737    2584 scope.go:117] "RemoveContainer" containerID="d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0"
Sep 16 19:13:55.763984 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:13:55.763968    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0\": container with ID starting with d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0 not found: ID does not exist" containerID="d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0"
Sep 16 19:13:55.764024 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.763988    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0"} err="failed to get container status \"d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0\": rpc error: code = NotFound desc = could not find container \"d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0\": container with ID starting with d683e733a28b0483829169de95ce2bc62d5dac6a6dcf315fb5c9e933e1923ef0 not found: ID does not exist"
Sep 16 19:13:55.827767 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.827749    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9c7k\" (UniqueName: \"kubernetes.io/projected/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-api-access-f9c7k\") pod \"f3caed81-90b9-4446-8c57-aa7b24ac3784\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") "
Sep 16 19:13:55.827860 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.827788    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3caed81-90b9-4446-8c57-aa7b24ac3784-proxy-tls\") pod \"f3caed81-90b9-4446-8c57-aa7b24ac3784\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") "
Sep 16 19:13:55.827860 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.827808    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3caed81-90b9-4446-8c57-aa7b24ac3784-kserve-provision-location\") pod \"f3caed81-90b9-4446-8c57-aa7b24ac3784\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") "
Sep 16 19:13:55.827860 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.827837    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-rbac-proxy-sar-config\") pod \"f3caed81-90b9-4446-8c57-aa7b24ac3784\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") "
Sep 16 19:13:55.827966 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.827875    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-cabundle-cert\") pod \"f3caed81-90b9-4446-8c57-aa7b24ac3784\" (UID: \"f3caed81-90b9-4446-8c57-aa7b24ac3784\") "
Sep 16 19:13:55.828208 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.828188    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3caed81-90b9-4446-8c57-aa7b24ac3784" (UID: "f3caed81-90b9-4446-8c57-aa7b24ac3784"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:55.828308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.828250    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3caed81-90b9-4446-8c57-aa7b24ac3784" (UID: "f3caed81-90b9-4446-8c57-aa7b24ac3784"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:55.828801 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.828779    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3caed81-90b9-4446-8c57-aa7b24ac3784-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3caed81-90b9-4446-8c57-aa7b24ac3784" (UID: "f3caed81-90b9-4446-8c57-aa7b24ac3784"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:55.829711 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.829693    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3caed81-90b9-4446-8c57-aa7b24ac3784-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3caed81-90b9-4446-8c57-aa7b24ac3784" (UID: "f3caed81-90b9-4446-8c57-aa7b24ac3784"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:55.829852 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.829832    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-api-access-f9c7k" (OuterVolumeSpecName: "kube-api-access-f9c7k") pod "f3caed81-90b9-4446-8c57-aa7b24ac3784" (UID: "f3caed81-90b9-4446-8c57-aa7b24ac3784"). InnerVolumeSpecName "kube-api-access-f9c7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:55.929350 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.929327    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9c7k\" (UniqueName: \"kubernetes.io/projected/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-api-access-f9c7k\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:55.929350 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.929349    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3caed81-90b9-4446-8c57-aa7b24ac3784-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:55.929471 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.929364    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3caed81-90b9-4446-8c57-aa7b24ac3784-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:55.929471 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.929378    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:55.929471 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:55.929391    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3caed81-90b9-4446-8c57-aa7b24ac3784-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:13:56.064287 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:56.064254    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"]
Sep 16 19:13:56.068501 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:56.068476    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-t9hmt"]
Sep 16 19:13:57.815715 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:57.815674    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" path="/var/lib/kubelet/pods/f3caed81-90b9-4446-8c57-aa7b24ac3784/volumes"
Sep 16 19:13:59.332761 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.332727    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"]
Sep 16 19:13:59.333238 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.333217    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kube-rbac-proxy"
Sep 16 19:13:59.333333 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.333242    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kube-rbac-proxy"
Sep 16 19:13:59.333333 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.333257    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="storage-initializer"
Sep 16 19:13:59.333333 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.333283    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="storage-initializer"
Sep 16 19:13:59.333333 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.333294    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kserve-container"
Sep 16 19:13:59.333333 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.333304    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kserve-container"
Sep 16 19:13:59.333590 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.333421    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kube-rbac-proxy"
Sep 16 19:13:59.333590 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.333435    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3caed81-90b9-4446-8c57-aa7b24ac3784" containerName="kserve-container"
Sep 16 19:13:59.338244 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.338228    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.343231 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.343212    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:13:59.343369 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.343277    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 16 19:13:59.343369 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.343288    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:13:59.343494 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.343407    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:13:59.343591 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.343573    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:13:59.344361 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.344344    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:13:59.344361 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.344353    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:13:59.349180 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.349155    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"]
Sep 16 19:13:59.353783 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.353755    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.353908 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.353803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7cd877-e483-4542-a919-80345bccc445-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.353908 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.353864    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.354038 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.353905    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6mm\" (UniqueName: \"kubernetes.io/projected/aa7cd877-e483-4542-a919-80345bccc445-kube-api-access-6l6mm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.354038 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.353948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7cd877-e483-4542-a919-80345bccc445-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.454988 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.454963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.455099 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.454995    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7cd877-e483-4542-a919-80345bccc445-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.455099 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.455020    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.455099 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.455044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6mm\" (UniqueName: \"kubernetes.io/projected/aa7cd877-e483-4542-a919-80345bccc445-kube-api-access-6l6mm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.455099 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.455076    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7cd877-e483-4542-a919-80345bccc445-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.455327 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:13:59.455154    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 16 19:13:59.455327 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:13:59.455220    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa7cd877-e483-4542-a919-80345bccc445-proxy-tls podName:aa7cd877-e483-4542-a919-80345bccc445 nodeName:}" failed. No retries permitted until 2026-09-16 19:13:59.955199252 +0000 UTC m=+1770.622138394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aa7cd877-e483-4542-a919-80345bccc445-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" (UID: "aa7cd877-e483-4542-a919-80345bccc445") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 16 19:13:59.455514 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.455497    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7cd877-e483-4542-a919-80345bccc445-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.455664 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.455645    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.455845 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.455824    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.463703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.463685    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6mm\" (UniqueName: \"kubernetes.io/projected/aa7cd877-e483-4542-a919-80345bccc445-kube-api-access-6l6mm\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.959188 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.959157    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7cd877-e483-4542-a919-80345bccc445-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:13:59.961467 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:13:59.961445    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7cd877-e483-4542-a919-80345bccc445-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:14:00.249226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:00.249148    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:14:00.522001 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:00.521977    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"]
Sep 16 19:14:00.524799 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:14:00.524771    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7cd877_e483_4542_a919_80345bccc445.slice/crio-6f97b14173a23dff518d79ca8ea3a4e86643c5ef09a36a72269d6d3c3f11b41a WatchSource:0}: Error finding container 6f97b14173a23dff518d79ca8ea3a4e86643c5ef09a36a72269d6d3c3f11b41a: Status 404 returned error can't find the container with id 6f97b14173a23dff518d79ca8ea3a4e86643c5ef09a36a72269d6d3c3f11b41a
Sep 16 19:14:00.770024 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:00.769980    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" event={"ID":"aa7cd877-e483-4542-a919-80345bccc445","Type":"ContainerStarted","Data":"3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e"}
Sep 16 19:14:00.770227 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:00.770070    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" event={"ID":"aa7cd877-e483-4542-a919-80345bccc445","Type":"ContainerStarted","Data":"6f97b14173a23dff518d79ca8ea3a4e86643c5ef09a36a72269d6d3c3f11b41a"}
Sep 16 19:14:09.817320 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:09.817225    2584 generic.go:356] "Generic (PLEG): container finished" podID="aa7cd877-e483-4542-a919-80345bccc445" containerID="3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e" exitCode=0
Sep 16 19:14:09.817320 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:09.817294    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" event={"ID":"aa7cd877-e483-4542-a919-80345bccc445","Type":"ContainerDied","Data":"3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e"}
Sep 16 19:14:10.823971 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:10.823933    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" event={"ID":"aa7cd877-e483-4542-a919-80345bccc445","Type":"ContainerStarted","Data":"f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f"}
Sep 16 19:14:10.824444 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:10.823999    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" event={"ID":"aa7cd877-e483-4542-a919-80345bccc445","Type":"ContainerStarted","Data":"bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59"}
Sep 16 19:14:10.824444 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:10.824181    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:14:10.844741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:10.844697    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" podStartSLOduration=3.006497048 podStartE2EDuration="11.84468383s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.838186782s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:10.842671096 +0000 UTC m=+1781.509610269" watchObservedRunningTime="2026-09-16 19:14:10.84468383 +0000 UTC m=+1781.511622977"
Sep 16 19:14:11.829353 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:11.829320    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:14:11.830904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:11.830871    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:14:12.833409 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:12.833374    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:14:17.837497 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:17.837462    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:14:17.838129 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:17.838097    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 16 19:14:27.838504 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:27.838476    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:14:34.406025 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:34.405994    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"]
Sep 16 19:14:34.406495 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:34.406316    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kserve-container" containerID="cri-o://bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59" gracePeriod=30
Sep 16 19:14:34.406495 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:34.406367    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kube-rbac-proxy" containerID="cri-o://f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f" gracePeriod=30
Sep 16 19:14:34.932259 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:34.932224    2584 generic.go:356] "Generic (PLEG): container finished" podID="aa7cd877-e483-4542-a919-80345bccc445" containerID="f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f" exitCode=2
Sep 16 19:14:34.932452 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:34.932311    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" event={"ID":"aa7cd877-e483-4542-a919-80345bccc445","Type":"ContainerDied","Data":"f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f"}
Sep 16 19:14:35.645920 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.645897    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:14:35.727892 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.727860    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-kube-rbac-proxy-sar-config\") pod \"aa7cd877-e483-4542-a919-80345bccc445\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") "
Sep 16 19:14:35.728040 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.727944    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7cd877-e483-4542-a919-80345bccc445-proxy-tls\") pod \"aa7cd877-e483-4542-a919-80345bccc445\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") "
Sep 16 19:14:35.728040 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.727963    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-cabundle-cert\") pod \"aa7cd877-e483-4542-a919-80345bccc445\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") "
Sep 16 19:14:35.728040 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.727980    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l6mm\" (UniqueName: \"kubernetes.io/projected/aa7cd877-e483-4542-a919-80345bccc445-kube-api-access-6l6mm\") pod \"aa7cd877-e483-4542-a919-80345bccc445\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") "
Sep 16 19:14:35.728040 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.728008    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7cd877-e483-4542-a919-80345bccc445-kserve-provision-location\") pod \"aa7cd877-e483-4542-a919-80345bccc445\" (UID: \"aa7cd877-e483-4542-a919-80345bccc445\") "
Sep 16 19:14:35.728381 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.728351    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa7cd877-e483-4542-a919-80345bccc445" (UID: "aa7cd877-e483-4542-a919-80345bccc445"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:35.728471 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.728376    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa7cd877-e483-4542-a919-80345bccc445" (UID: "aa7cd877-e483-4542-a919-80345bccc445"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:35.729115 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.729090    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7cd877-e483-4542-a919-80345bccc445-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa7cd877-e483-4542-a919-80345bccc445" (UID: "aa7cd877-e483-4542-a919-80345bccc445"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:35.730091 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.730069    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7cd877-e483-4542-a919-80345bccc445-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa7cd877-e483-4542-a919-80345bccc445" (UID: "aa7cd877-e483-4542-a919-80345bccc445"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:35.730159 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.730111    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7cd877-e483-4542-a919-80345bccc445-kube-api-access-6l6mm" (OuterVolumeSpecName: "kube-api-access-6l6mm") pod "aa7cd877-e483-4542-a919-80345bccc445" (UID: "aa7cd877-e483-4542-a919-80345bccc445"). InnerVolumeSpecName "kube-api-access-6l6mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:35.828444 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.828422    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7cd877-e483-4542-a919-80345bccc445-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:14:35.828444 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.828442    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:14:35.828568 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.828452    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l6mm\" (UniqueName: \"kubernetes.io/projected/aa7cd877-e483-4542-a919-80345bccc445-kube-api-access-6l6mm\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:14:35.828568 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.828461    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7cd877-e483-4542-a919-80345bccc445-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:14:35.828568 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.828469    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7cd877-e483-4542-a919-80345bccc445-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:14:35.938458 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.938426    2584 generic.go:356] "Generic (PLEG): container finished" podID="aa7cd877-e483-4542-a919-80345bccc445" containerID="bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59" exitCode=0
Sep 16 19:14:35.938593 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.938510    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"
Sep 16 19:14:35.938593 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.938518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" event={"ID":"aa7cd877-e483-4542-a919-80345bccc445","Type":"ContainerDied","Data":"bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59"}
Sep 16 19:14:35.938712 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.938598    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz" event={"ID":"aa7cd877-e483-4542-a919-80345bccc445","Type":"ContainerDied","Data":"6f97b14173a23dff518d79ca8ea3a4e86643c5ef09a36a72269d6d3c3f11b41a"}
Sep 16 19:14:35.938712 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.938640    2584 scope.go:117] "RemoveContainer" containerID="f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f"
Sep 16 19:14:35.946984 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.946961    2584 scope.go:117] "RemoveContainer" containerID="bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59"
Sep 16 19:14:35.954143 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.954128    2584 scope.go:117] "RemoveContainer" containerID="3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e"
Sep 16 19:14:35.957162 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.957141    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"]
Sep 16 19:14:35.962010 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.961990    2584 scope.go:117] "RemoveContainer" containerID="f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f"
Sep 16 19:14:35.962357 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:14:35.962334    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f\": container with ID starting with f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f not found: ID does not exist" containerID="f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f"
Sep 16 19:14:35.962442 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.962369    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f"} err="failed to get container status \"f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f\": rpc error: code = NotFound desc = could not find container \"f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f\": container with ID starting with f4368178e161023a39f8eb217c9219e881c1fcbf768bd4a59705abd70039838f not found: ID does not exist"
Sep 16 19:14:35.962442 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.962384    2584 scope.go:117] "RemoveContainer" containerID="bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59"
Sep 16 19:14:35.962653 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:14:35.962638    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59\": container with ID starting with bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59 not found: ID does not exist" containerID="bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59"
Sep 16 19:14:35.962699 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.962669    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59"} err="failed to get container status \"bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59\": rpc error: code = NotFound desc = could not find container \"bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59\": container with ID starting with bfdd68668e13689dad88116db2ebe47b3873f48b7fc536944257f7d6a71d6e59 not found: ID does not exist"
Sep 16 19:14:35.962699 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.962681    2584 scope.go:117] "RemoveContainer" containerID="3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e"
Sep 16 19:14:35.962951 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:14:35.962932    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e\": container with ID starting with 3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e not found: ID does not exist" containerID="3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e"
Sep 16 19:14:35.963006 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.962956    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e"} err="failed to get container status \"3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e\": rpc error: code = NotFound desc = could not find container \"3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e\": container with ID starting with 3516f5901af3ae32f98d241923d834b95e234e53191aebfbff82bfb0b195618e not found: ID does not exist"
Sep 16 19:14:35.963794 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:35.963775    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-rw4xz"]
Sep 16 19:14:37.816983 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:37.816953    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7cd877-e483-4542-a919-80345bccc445" path="/var/lib/kubelet/pods/aa7cd877-e483-4542-a919-80345bccc445/volumes"
Sep 16 19:14:39.680919 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.680885    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"]
Sep 16 19:14:39.681303 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.681231    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kube-rbac-proxy"
Sep 16 19:14:39.681303 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.681244    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kube-rbac-proxy"
Sep 16 19:14:39.681303 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.681253    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kserve-container"
Sep 16 19:14:39.681303 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.681258    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kserve-container"
Sep 16 19:14:39.681303 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.681287    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="storage-initializer"
Sep 16 19:14:39.681303 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.681297    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="storage-initializer"
Sep 16 19:14:39.681507 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.681367    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kserve-container"
Sep 16 19:14:39.681507 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.681374    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7cd877-e483-4542-a919-80345bccc445" containerName="kube-rbac-proxy"
Sep 16 19:14:39.686162 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.686133    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.690013 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.689988    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:14:39.690179 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.690017    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:14:39.690179 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.690128    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:14:39.690179 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.690142    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:14:39.690360 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.690238    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:39.690404 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.690360    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:14:39.690404 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.690383    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:39.700501 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.700482    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"]
Sep 16 19:14:39.758812 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.758785    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.758920 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.758842    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hf8k\" (UniqueName: \"kubernetes.io/projected/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-api-access-2hf8k\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.758920 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.758888    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.758920 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.758909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c863b50-666d-4ac8-9e0e-2007d889d4e2-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.759026 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.758933    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.859463 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.859431    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hf8k\" (UniqueName: \"kubernetes.io/projected/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-api-access-2hf8k\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.859607 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.859471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.859607 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.859497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c863b50-666d-4ac8-9e0e-2007d889d4e2-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.859607 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.859533    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.859607 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.859579    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.859979 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.859957    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.860134 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.860115    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.860220 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.860201    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.861929 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.861907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c863b50-666d-4ac8-9e0e-2007d889d4e2-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.867728 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.867706    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hf8k\" (UniqueName: \"kubernetes.io/projected/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-api-access-2hf8k\") pod \"isvc-canary-create-predictor-6fb974599-lx7v5\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:39.996896 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:39.996842    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:40.280716 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:40.280691    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"]
Sep 16 19:14:40.282839 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:14:40.282804    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c863b50_666d_4ac8_9e0e_2007d889d4e2.slice/crio-f8a50a70e9ec1dfb6c7f09dbba074bd9ff0e63a8a55d54b87ae8d3e141412871 WatchSource:0}: Error finding container f8a50a70e9ec1dfb6c7f09dbba074bd9ff0e63a8a55d54b87ae8d3e141412871: Status 404 returned error can't find the container with id f8a50a70e9ec1dfb6c7f09dbba074bd9ff0e63a8a55d54b87ae8d3e141412871
Sep 16 19:14:40.963130 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:40.963095    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" event={"ID":"4c863b50-666d-4ac8-9e0e-2007d889d4e2","Type":"ContainerStarted","Data":"ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85"}
Sep 16 19:14:40.963539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:40.963179    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" event={"ID":"4c863b50-666d-4ac8-9e0e-2007d889d4e2","Type":"ContainerStarted","Data":"f8a50a70e9ec1dfb6c7f09dbba074bd9ff0e63a8a55d54b87ae8d3e141412871"}
Sep 16 19:14:44.982343 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:44.982310    2584 generic.go:356] "Generic (PLEG): container finished" podID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerID="ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85" exitCode=0
Sep 16 19:14:44.982711 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:44.982350    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" event={"ID":"4c863b50-666d-4ac8-9e0e-2007d889d4e2","Type":"ContainerDied","Data":"ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85"}
Sep 16 19:14:44.983449 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:44.983431    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:14:59.064308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:59.064254    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" event={"ID":"4c863b50-666d-4ac8-9e0e-2007d889d4e2","Type":"ContainerStarted","Data":"6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2"}
Sep 16 19:14:59.064749 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:59.064349    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" event={"ID":"4c863b50-666d-4ac8-9e0e-2007d889d4e2","Type":"ContainerStarted","Data":"47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a"}
Sep 16 19:14:59.064749 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:59.064539    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:14:59.090138 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:14:59.090095    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podStartSLOduration=3.02092852 podStartE2EDuration="20.090083729s" totalImagesPullingTime="13.152920263s" totalInitContainerRuntime="3.916234946s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:59.08831873 +0000 UTC m=+1829.755257895" watchObservedRunningTime="2026-09-16 19:14:59.090083729 +0000 UTC m=+1829.757022875"
Sep 16 19:15:00.069166 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:00.069131    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:15:00.070562 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:00.070538    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:15:01.073654 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:01.073612    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:15:06.078386 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:06.078351    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:15:06.078972 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:06.078946    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:15:16.078965 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:16.078920    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:15:26.079588 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:26.079548    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:15:36.079590 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:36.079548    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:15:46.079748 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:46.079703    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:15:56.079281 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:15:56.079035    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:16:06.079749 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:06.079709    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 16 19:16:16.080352 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:16.080322    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:16:19.675943 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:19.675907    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"]
Sep 16 19:16:19.676317 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:19.676191    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container" containerID="cri-o://47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a" gracePeriod=30
Sep 16 19:16:19.676317 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:19.676283    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kube-rbac-proxy" containerID="cri-o://6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2" gracePeriod=30
Sep 16 19:16:20.426560 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:20.426511    2584 generic.go:356] "Generic (PLEG): container finished" podID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerID="6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2" exitCode=2
Sep 16 19:16:20.426724 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:20.426568    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" event={"ID":"4c863b50-666d-4ac8-9e0e-2007d889d4e2","Type":"ContainerDied","Data":"6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2"}
Sep 16 19:16:21.073883 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:21.073842    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 16 19:16:24.017542 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.017517    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:16:24.168858 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.168828    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-rbac-proxy-sar-config\") pod \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") "
Sep 16 19:16:24.169062 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.168872    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c863b50-666d-4ac8-9e0e-2007d889d4e2-proxy-tls\") pod \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") "
Sep 16 19:16:24.169062 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.168901    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hf8k\" (UniqueName: \"kubernetes.io/projected/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-api-access-2hf8k\") pod \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") "
Sep 16 19:16:24.169062 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.168925    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kserve-provision-location\") pod \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") "
Sep 16 19:16:24.169062 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.168942    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-cabundle-cert\") pod \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\" (UID: \"4c863b50-666d-4ac8-9e0e-2007d889d4e2\") "
Sep 16 19:16:24.169319 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.169247    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c863b50-666d-4ac8-9e0e-2007d889d4e2" (UID: "4c863b50-666d-4ac8-9e0e-2007d889d4e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:24.169379 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.169298    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4c863b50-666d-4ac8-9e0e-2007d889d4e2" (UID: "4c863b50-666d-4ac8-9e0e-2007d889d4e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:24.169467 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.169443    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4c863b50-666d-4ac8-9e0e-2007d889d4e2" (UID: "4c863b50-666d-4ac8-9e0e-2007d889d4e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:24.171063 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.171042    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-api-access-2hf8k" (OuterVolumeSpecName: "kube-api-access-2hf8k") pod "4c863b50-666d-4ac8-9e0e-2007d889d4e2" (UID: "4c863b50-666d-4ac8-9e0e-2007d889d4e2"). InnerVolumeSpecName "kube-api-access-2hf8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:24.171143 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.171103    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c863b50-666d-4ac8-9e0e-2007d889d4e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c863b50-666d-4ac8-9e0e-2007d889d4e2" (UID: "4c863b50-666d-4ac8-9e0e-2007d889d4e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:24.269631 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.269595    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.269631 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.269627    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c863b50-666d-4ac8-9e0e-2007d889d4e2-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.269631 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.269638    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hf8k\" (UniqueName: \"kubernetes.io/projected/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kube-api-access-2hf8k\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.269870 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.269647    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c863b50-666d-4ac8-9e0e-2007d889d4e2-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.269870 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.269657    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c863b50-666d-4ac8-9e0e-2007d889d4e2-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.447201 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.447115    2584 generic.go:356] "Generic (PLEG): container finished" podID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerID="47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a" exitCode=0
Sep 16 19:16:24.447201 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.447162    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"
Sep 16 19:16:24.447394 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.447162    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" event={"ID":"4c863b50-666d-4ac8-9e0e-2007d889d4e2","Type":"ContainerDied","Data":"47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a"}
Sep 16 19:16:24.447394 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.447315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5" event={"ID":"4c863b50-666d-4ac8-9e0e-2007d889d4e2","Type":"ContainerDied","Data":"f8a50a70e9ec1dfb6c7f09dbba074bd9ff0e63a8a55d54b87ae8d3e141412871"}
Sep 16 19:16:24.447394 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.447344    2584 scope.go:117] "RemoveContainer" containerID="6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2"
Sep 16 19:16:24.455874 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.455856    2584 scope.go:117] "RemoveContainer" containerID="47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a"
Sep 16 19:16:24.463064 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.463046    2584 scope.go:117] "RemoveContainer" containerID="ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85"
Sep 16 19:16:24.470130 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.470107    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"]
Sep 16 19:16:24.471598 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.471582    2584 scope.go:117] "RemoveContainer" containerID="6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2"
Sep 16 19:16:24.471880 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:16:24.471861    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2\": container with ID starting with 6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2 not found: ID does not exist" containerID="6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2"
Sep 16 19:16:24.471951 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.471887    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2"} err="failed to get container status \"6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2\": rpc error: code = NotFound desc = could not find container \"6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2\": container with ID starting with 6b5c49e751ae80c20b1ecefc4773194d10334037fe75cbfa6ebdb86aa6b83bb2 not found: ID does not exist"
Sep 16 19:16:24.471951 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.471908    2584 scope.go:117] "RemoveContainer" containerID="47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a"
Sep 16 19:16:24.472167 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:16:24.472145    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a\": container with ID starting with 47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a not found: ID does not exist" containerID="47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a"
Sep 16 19:16:24.472228 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.472176    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a"} err="failed to get container status \"47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a\": rpc error: code = NotFound desc = could not find container \"47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a\": container with ID starting with 47bfc8b0f32b7a706b697836dde79b27ea495df68473bd4ad9dc5a016944078a not found: ID does not exist"
Sep 16 19:16:24.472228 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.472201    2584 scope.go:117] "RemoveContainer" containerID="ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85"
Sep 16 19:16:24.472533 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:16:24.472517    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85\": container with ID starting with ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85 not found: ID does not exist" containerID="ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85"
Sep 16 19:16:24.472582 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.472537    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85"} err="failed to get container status \"ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85\": rpc error: code = NotFound desc = could not find container \"ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85\": container with ID starting with ba54c2b0c56ea64ff0ec05ed10b7e7d9a80115538bad48470a6bcc31da8fdf85 not found: ID does not exist"
Sep 16 19:16:24.474102 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.474084    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-lx7v5"]
Sep 16 19:16:24.880618 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.880540    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"]
Sep 16 19:16:24.880865 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.880852    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container"
Sep 16 19:16:24.880928 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.880866    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container"
Sep 16 19:16:24.880928 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.880877    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="storage-initializer"
Sep 16 19:16:24.880928 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.880882    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="storage-initializer"
Sep 16 19:16:24.880928 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.880897    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kube-rbac-proxy"
Sep 16 19:16:24.880928 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.880903    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kube-rbac-proxy"
Sep 16 19:16:24.881216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.880957    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kserve-container"
Sep 16 19:16:24.881216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.880966    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" containerName="kube-rbac-proxy"
Sep 16 19:16:24.885817 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.885798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:24.888793 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.888770    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:16:24.889587 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.889542    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:16:24.889723 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.889609    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:16:24.889791 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.889776    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:16:24.890930 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.890912    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:16:24.891043 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.890912    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:16:24.891043 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.890965    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:16:24.897547 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.897526    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"]
Sep 16 19:16:24.974803 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.974774    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:24.974927 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.974856    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rbz\" (UniqueName: \"kubernetes.io/projected/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-api-access-p6rbz\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:24.974927 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.974887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:24.975024 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.974924    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:24.975024 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.974992    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:24.993025 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.993003    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"]
Sep 16 19:16:24.996707 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.996688    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:24.999392 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:24.999373    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:16:25.011666 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.011643    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"]
Sep 16 19:16:25.076101 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076070    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.076544 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.076544 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076137    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f641c9b8-93f2-4504-a371-68d9639d34c5-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.076544 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076194    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.076544 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f641c9b8-93f2-4504-a371-68d9639d34c5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.076544 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076257    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.076544 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.076544 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076397    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgskb\" (UniqueName: \"kubernetes.io/projected/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-api-access-bgskb\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.076544 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076437    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rbz\" (UniqueName: \"kubernetes.io/projected/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-api-access-p6rbz\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.076544 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076467    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.076957 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076802    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.076957 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076898    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.077031 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.076955    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.079304 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.079250    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.085310 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.085261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rbz\" (UniqueName: \"kubernetes.io/projected/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-api-access-p6rbz\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.177734 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.177702    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f641c9b8-93f2-4504-a371-68d9639d34c5-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.177904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.177758    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.177904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.177781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f641c9b8-93f2-4504-a371-68d9639d34c5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.177904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.177798    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.177904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.177873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgskb\" (UniqueName: \"kubernetes.io/projected/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-api-access-bgskb\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.178304 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.178260    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f641c9b8-93f2-4504-a371-68d9639d34c5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.178582 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.178561    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.178582 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.178576    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.180228 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.180202    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f641c9b8-93f2-4504-a371-68d9639d34c5-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.185953 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.185931    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgskb\" (UniqueName: \"kubernetes.io/projected/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-api-access-bgskb\") pod \"isvc-canary-promote-predictor-649dd98487-wprh7\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.197691 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.197671    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:25.306845 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.306805    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:25.469898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.469873    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"]
Sep 16 19:16:25.471675 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:16:25.471648    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1840b2e4_9d55_4bbb_a1c8_1ec3a35aba5e.slice/crio-ffbf8e1a18e6c4476d3dafb2dfd95f8587ab9cb2281441ebac2ef55bb7bca66c WatchSource:0}: Error finding container ffbf8e1a18e6c4476d3dafb2dfd95f8587ab9cb2281441ebac2ef55bb7bca66c: Status 404 returned error can't find the container with id ffbf8e1a18e6c4476d3dafb2dfd95f8587ab9cb2281441ebac2ef55bb7bca66c
Sep 16 19:16:25.581934 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.581906    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"]
Sep 16 19:16:25.583797 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:16:25.583764    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf641c9b8_93f2_4504_a371_68d9639d34c5.slice/crio-ed947ed17627add1f131ef88960701de850d2bcd9b4618f38ebc6a33238df3d1 WatchSource:0}: Error finding container ed947ed17627add1f131ef88960701de850d2bcd9b4618f38ebc6a33238df3d1: Status 404 returned error can't find the container with id ed947ed17627add1f131ef88960701de850d2bcd9b4618f38ebc6a33238df3d1
Sep 16 19:16:25.820431 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:25.820344    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c863b50-666d-4ac8-9e0e-2007d889d4e2" path="/var/lib/kubelet/pods/4c863b50-666d-4ac8-9e0e-2007d889d4e2/volumes"
Sep 16 19:16:26.459756 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:26.459714    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" event={"ID":"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e","Type":"ContainerStarted","Data":"d8e0735f09ad0b4bd07c55d4a393dbe97e520e68245091bd977cc525e61e1ac1"}
Sep 16 19:16:26.460197 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:26.459789    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" event={"ID":"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e","Type":"ContainerStarted","Data":"ffbf8e1a18e6c4476d3dafb2dfd95f8587ab9cb2281441ebac2ef55bb7bca66c"}
Sep 16 19:16:26.461193 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:26.461169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" event={"ID":"f641c9b8-93f2-4504-a371-68d9639d34c5","Type":"ContainerStarted","Data":"af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31"}
Sep 16 19:16:26.461325 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:26.461216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" event={"ID":"f641c9b8-93f2-4504-a371-68d9639d34c5","Type":"ContainerStarted","Data":"ed947ed17627add1f131ef88960701de850d2bcd9b4618f38ebc6a33238df3d1"}
Sep 16 19:16:29.476606 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:29.476574    2584 generic.go:356] "Generic (PLEG): container finished" podID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerID="d8e0735f09ad0b4bd07c55d4a393dbe97e520e68245091bd977cc525e61e1ac1" exitCode=0
Sep 16 19:16:29.476606 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:29.476612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" event={"ID":"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e","Type":"ContainerDied","Data":"d8e0735f09ad0b4bd07c55d4a393dbe97e520e68245091bd977cc525e61e1ac1"}
Sep 16 19:16:30.484147 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:30.484107    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" event={"ID":"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e","Type":"ContainerStarted","Data":"0feaea79b66fbafadb440f52d39594414021494d28236c87b5c1291cb583efd7"}
Sep 16 19:16:30.484620 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:30.484182    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" event={"ID":"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e","Type":"ContainerStarted","Data":"e2e95c084e512391084468d1c381ff4be254d1beef07ec5fb79a4f3b284f7472"}
Sep 16 19:16:30.484620 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:30.484387    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:30.485679 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:30.485656    2584 generic.go:356] "Generic (PLEG): container finished" podID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerID="af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31" exitCode=0
Sep 16 19:16:30.485782 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:30.485730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" event={"ID":"f641c9b8-93f2-4504-a371-68d9639d34c5","Type":"ContainerDied","Data":"af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31"}
Sep 16 19:16:30.506445 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:30.506397    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podStartSLOduration=2.692451621 podStartE2EDuration="6.506378241s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81392662s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:30.504143309 +0000 UTC m=+1921.171082513" watchObservedRunningTime="2026-09-16 19:16:30.506378241 +0000 UTC m=+1921.173317394"
Sep 16 19:16:31.492617 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:31.492578    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" event={"ID":"f641c9b8-93f2-4504-a371-68d9639d34c5","Type":"ContainerStarted","Data":"0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d"}
Sep 16 19:16:31.493044 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:31.492664    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" event={"ID":"f641c9b8-93f2-4504-a371-68d9639d34c5","Type":"ContainerStarted","Data":"ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839"}
Sep 16 19:16:31.493044 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:31.492805    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:31.494212 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:31.494185    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:16:31.515031 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:31.514977    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podStartSLOduration=3.709334606 podStartE2EDuration="7.514962947s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.805628341s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:31.513360781 +0000 UTC m=+1922.180299948" watchObservedRunningTime="2026-09-16 19:16:31.514962947 +0000 UTC m=+1922.181902094"
Sep 16 19:16:32.497098 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:32.497053    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:16:36.493766 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:36.493735    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:36.494211 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:36.494190    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:36.495335 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:36.495299    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:16:36.498780 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:36.498761    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:16:36.518339 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:36.518313    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:16:37.502663 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:37.502634    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:16:37.503098 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:37.503069    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:16:37.523190 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:37.523158    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:16:47.503292 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:47.503225    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:16:47.523732 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:47.523692    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:16:57.503237 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:57.503195    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:16:57.523974 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:16:57.523935    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:17:07.503205 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:07.503163    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:17:07.523380 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:07.523346    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:17:17.503167 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:17.503126    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:17:17.523572 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:17.523531    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:17:25.768685 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.768612    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"]
Sep 16 19:17:25.772258 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.772238    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.774819 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.774796    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 16 19:17:25.774913 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.774864    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 16 19:17:25.784906 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.784884    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"]
Sep 16 19:17:25.862574 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.862543    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.862703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.862579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.862703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.862651    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.862703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.862683    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.862703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.862699    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mdb\" (UniqueName: \"kubernetes.io/projected/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-api-access-d2mdb\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.964049 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.964019    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.964175 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.964054    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.964175 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.964070    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mdb\" (UniqueName: \"kubernetes.io/projected/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-api-access-d2mdb\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.964175 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.964107    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.964175 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.964137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.964559 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.964533    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.964766 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.964744    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.964818 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.964800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.966496 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.966480    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:25.973131 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:25.973108    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mdb\" (UniqueName: \"kubernetes.io/projected/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-api-access-d2mdb\") pod \"isvc-tensorflow-predictor-547dfc5669-xqjk6\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:26.085404 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:26.085346    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:26.360011 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:26.359986    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"]
Sep 16 19:17:26.361394 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:17:26.361362    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee32aa7e_1e71_4965_b484_ef3e2aa8518e.slice/crio-125ba776bac64482dc6e392f7c5b7576788943c80f42cc268fa97744b3fa3e72 WatchSource:0}: Error finding container 125ba776bac64482dc6e392f7c5b7576788943c80f42cc268fa97744b3fa3e72: Status 404 returned error can't find the container with id 125ba776bac64482dc6e392f7c5b7576788943c80f42cc268fa97744b3fa3e72
Sep 16 19:17:26.752949 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:26.752913    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" event={"ID":"ee32aa7e-1e71-4965-b484-ef3e2aa8518e","Type":"ContainerStarted","Data":"4db8b4134525013b3be7c5d4b4eec9b82141c5a923659cdcb1be54bebe09cb6d"}
Sep 16 19:17:26.753132 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:26.752981    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" event={"ID":"ee32aa7e-1e71-4965-b484-ef3e2aa8518e","Type":"ContainerStarted","Data":"125ba776bac64482dc6e392f7c5b7576788943c80f42cc268fa97744b3fa3e72"}
Sep 16 19:17:27.503703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:27.503612    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:17:27.524024 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:27.523985    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:17:31.783242 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:31.783208    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerID="4db8b4134525013b3be7c5d4b4eec9b82141c5a923659cdcb1be54bebe09cb6d" exitCode=0
Sep 16 19:17:31.783631 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:31.783294    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" event={"ID":"ee32aa7e-1e71-4965-b484-ef3e2aa8518e","Type":"ContainerDied","Data":"4db8b4134525013b3be7c5d4b4eec9b82141c5a923659cdcb1be54bebe09cb6d"}
Sep 16 19:17:35.816857 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:35.816826    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" event={"ID":"ee32aa7e-1e71-4965-b484-ef3e2aa8518e","Type":"ContainerStarted","Data":"d134c76039253cabe07750990c60001e580714e6d841e72a2960f49864a69d8b"}
Sep 16 19:17:35.817247 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:35.816881    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" event={"ID":"ee32aa7e-1e71-4965-b484-ef3e2aa8518e","Type":"ContainerStarted","Data":"16016f3571d1a3b9a118e435dd6c3a3daa3cb61539505f0e1bd4e30395dc0e17"}
Sep 16 19:17:35.817247 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:35.817146    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:35.817359 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:35.817287    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:35.818772 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:35.818741    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:17:35.837624 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:35.837582    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podStartSLOduration=2.645411844 podStartE2EDuration="10.837570083s" totalImagesPullingTime="3.556356296s" totalInitContainerRuntime="4.635801943s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:35.835743405 +0000 UTC m=+1986.502682582" watchObservedRunningTime="2026-09-16 19:17:35.837570083 +0000 UTC m=+1986.504509229"
Sep 16 19:17:36.821255 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:36.821215    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:17:37.503924 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:37.503880    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:17:37.523226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:37.523188    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:17:37.825714 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:37.825633    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:17:42.829863 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:42.829832    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:42.830332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:42.830304    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 16 19:17:46.811905 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:46.811875    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:17:47.524001 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:47.523970    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:17:52.830763 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:52.830733    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:17:54.970244 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:54.970214    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"]
Sep 16 19:17:54.970676 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:54.970450    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" containerID="cri-o://ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839" gracePeriod=30
Sep 16 19:17:54.970676 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:54.970481    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kube-rbac-proxy" containerID="cri-o://0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d" gracePeriod=30
Sep 16 19:17:55.906537 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:55.906497    2584 generic.go:356] "Generic (PLEG): container finished" podID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerID="0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d" exitCode=2
Sep 16 19:17:55.906696 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:55.906550    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" event={"ID":"f641c9b8-93f2-4504-a371-68d9639d34c5","Type":"ContainerDied","Data":"0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d"}
Sep 16 19:17:56.499552 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:56.499511    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 16 19:17:57.523702 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:57.523665    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 16 19:17:59.418537 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.418506    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:17:59.452806 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.452779    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-cabundle-cert\") pod \"f641c9b8-93f2-4504-a371-68d9639d34c5\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") "
Sep 16 19:17:59.452982 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.452842    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-rbac-proxy-sar-config\") pod \"f641c9b8-93f2-4504-a371-68d9639d34c5\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") "
Sep 16 19:17:59.452982 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.452971    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f641c9b8-93f2-4504-a371-68d9639d34c5-proxy-tls\") pod \"f641c9b8-93f2-4504-a371-68d9639d34c5\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") "
Sep 16 19:17:59.453112 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.453008    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f641c9b8-93f2-4504-a371-68d9639d34c5-kserve-provision-location\") pod \"f641c9b8-93f2-4504-a371-68d9639d34c5\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") "
Sep 16 19:17:59.453112 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.453025    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgskb\" (UniqueName: \"kubernetes.io/projected/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-api-access-bgskb\") pod \"f641c9b8-93f2-4504-a371-68d9639d34c5\" (UID: \"f641c9b8-93f2-4504-a371-68d9639d34c5\") "
Sep 16 19:17:59.453218 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.453149    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f641c9b8-93f2-4504-a371-68d9639d34c5" (UID: "f641c9b8-93f2-4504-a371-68d9639d34c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:59.453218 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.453189    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f641c9b8-93f2-4504-a371-68d9639d34c5" (UID: "f641c9b8-93f2-4504-a371-68d9639d34c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:59.453414 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.453376    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f641c9b8-93f2-4504-a371-68d9639d34c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f641c9b8-93f2-4504-a371-68d9639d34c5" (UID: "f641c9b8-93f2-4504-a371-68d9639d34c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:59.453414 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.453395    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:17:59.453511 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.453417    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f641c9b8-93f2-4504-a371-68d9639d34c5-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:17:59.455167 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.455141    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-api-access-bgskb" (OuterVolumeSpecName: "kube-api-access-bgskb") pod "f641c9b8-93f2-4504-a371-68d9639d34c5" (UID: "f641c9b8-93f2-4504-a371-68d9639d34c5"). InnerVolumeSpecName "kube-api-access-bgskb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:59.455167 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.455158    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f641c9b8-93f2-4504-a371-68d9639d34c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f641c9b8-93f2-4504-a371-68d9639d34c5" (UID: "f641c9b8-93f2-4504-a371-68d9639d34c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:59.554725 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.554688    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f641c9b8-93f2-4504-a371-68d9639d34c5-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:17:59.554725 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.554719    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgskb\" (UniqueName: \"kubernetes.io/projected/f641c9b8-93f2-4504-a371-68d9639d34c5-kube-api-access-bgskb\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:17:59.554725 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.554729    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f641c9b8-93f2-4504-a371-68d9639d34c5-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:17:59.927822 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.927784    2584 generic.go:356] "Generic (PLEG): container finished" podID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerID="ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839" exitCode=0
Sep 16 19:17:59.928000 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.927837    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"
Sep 16 19:17:59.928000 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.927866    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" event={"ID":"f641c9b8-93f2-4504-a371-68d9639d34c5","Type":"ContainerDied","Data":"ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839"}
Sep 16 19:17:59.928000 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.927931    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7" event={"ID":"f641c9b8-93f2-4504-a371-68d9639d34c5","Type":"ContainerDied","Data":"ed947ed17627add1f131ef88960701de850d2bcd9b4618f38ebc6a33238df3d1"}
Sep 16 19:17:59.928000 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.927959    2584 scope.go:117] "RemoveContainer" containerID="0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d"
Sep 16 19:17:59.936555 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.936537    2584 scope.go:117] "RemoveContainer" containerID="ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839"
Sep 16 19:17:59.943749 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.943735    2584 scope.go:117] "RemoveContainer" containerID="af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31"
Sep 16 19:17:59.948545 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.948521    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"]
Sep 16 19:17:59.951390 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.951374    2584 scope.go:117] "RemoveContainer" containerID="0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d"
Sep 16 19:17:59.951686 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:17:59.951671    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d\": container with ID starting with 0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d not found: ID does not exist" containerID="0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d"
Sep 16 19:17:59.951737 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.951695    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d"} err="failed to get container status \"0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d\": rpc error: code = NotFound desc = could not find container \"0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d\": container with ID starting with 0c9f569d95ef33e82c35ae6a92199f9f63a4ac08fea550ca18cce1662851c62d not found: ID does not exist"
Sep 16 19:17:59.951737 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.951711    2584 scope.go:117] "RemoveContainer" containerID="ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839"
Sep 16 19:17:59.951930 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:17:59.951915    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839\": container with ID starting with ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839 not found: ID does not exist" containerID="ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839"
Sep 16 19:17:59.952008 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.951935    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839"} err="failed to get container status \"ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839\": rpc error: code = NotFound desc = could not find container \"ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839\": container with ID starting with ef58a14a6396cd4a71e7e81feda7c09cfffbaea1fcb76cac398f8bb33eeb4839 not found: ID does not exist"
Sep 16 19:17:59.952008 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.951947    2584 scope.go:117] "RemoveContainer" containerID="af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31"
Sep 16 19:17:59.952306 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:17:59.952241    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31\": container with ID starting with af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31 not found: ID does not exist" containerID="af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31"
Sep 16 19:17:59.952306 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.952291    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31"} err="failed to get container status \"af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31\": rpc error: code = NotFound desc = could not find container \"af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31\": container with ID starting with af073fa9d36cb3332895ecf6a16018eddb645a87ef941cb117ccfeef03a8ac31 not found: ID does not exist"
Sep 16 19:17:59.954637 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:17:59.954618    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-wprh7"]
Sep 16 19:18:01.816470 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:01.816432    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" path="/var/lib/kubelet/pods/f641c9b8-93f2-4504-a371-68d9639d34c5/volumes"
Sep 16 19:18:05.931624 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:05.931593    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"]
Sep 16 19:18:05.932082 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:05.931812    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kserve-container" containerID="cri-o://16016f3571d1a3b9a118e435dd6c3a3daa3cb61539505f0e1bd4e30395dc0e17" gracePeriod=30
Sep 16 19:18:05.932082 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:05.931857    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy" containerID="cri-o://d134c76039253cabe07750990c60001e580714e6d841e72a2960f49864a69d8b" gracePeriod=30
Sep 16 19:18:05.972717 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:18:05.972689    2584 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee32aa7e_1e71_4965_b484_ef3e2aa8518e.slice/crio-d134c76039253cabe07750990c60001e580714e6d841e72a2960f49864a69d8b.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 19:18:06.963495 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:06.963462    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerID="d134c76039253cabe07750990c60001e580714e6d841e72a2960f49864a69d8b" exitCode=2
Sep 16 19:18:06.963851 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:06.963536    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" event={"ID":"ee32aa7e-1e71-4965-b484-ef3e2aa8518e","Type":"ContainerDied","Data":"d134c76039253cabe07750990c60001e580714e6d841e72a2960f49864a69d8b"}
Sep 16 19:18:07.825923 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:07.825882    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 16 19:18:12.826586 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:12.826544    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 16 19:18:17.826812 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:17.826770    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 16 19:18:17.827316 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:17.826881    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:18:22.826465 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:22.826421    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 16 19:18:27.826648 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:27.826607    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 16 19:18:32.826363 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:32.826323    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 16 19:18:36.095339 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.095308    2584 generic.go:356] "Generic (PLEG): container finished" podID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerID="16016f3571d1a3b9a118e435dd6c3a3daa3cb61539505f0e1bd4e30395dc0e17" exitCode=137
Sep 16 19:18:36.095664 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.095382    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" event={"ID":"ee32aa7e-1e71-4965-b484-ef3e2aa8518e","Type":"ContainerDied","Data":"16016f3571d1a3b9a118e435dd6c3a3daa3cb61539505f0e1bd4e30395dc0e17"}
Sep 16 19:18:36.583041 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.583015    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:18:36.730525 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.730448    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-proxy-tls\") pod \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") "
Sep 16 19:18:36.730525 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.730498    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kserve-provision-location\") pod \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") "
Sep 16 19:18:36.730698 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.730526    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2mdb\" (UniqueName: \"kubernetes.io/projected/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-api-access-d2mdb\") pod \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") "
Sep 16 19:18:36.730750 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.730727    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-cabundle-cert\") pod \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") "
Sep 16 19:18:36.730798 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.730772    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-rbac-proxy-sar-config\") pod \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\" (UID: \"ee32aa7e-1e71-4965-b484-ef3e2aa8518e\") "
Sep 16 19:18:36.731136 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.731106    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee32aa7e-1e71-4965-b484-ef3e2aa8518e" (UID: "ee32aa7e-1e71-4965-b484-ef3e2aa8518e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:36.731252 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.731177    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee32aa7e-1e71-4965-b484-ef3e2aa8518e" (UID: "ee32aa7e-1e71-4965-b484-ef3e2aa8518e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:36.732726 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.732695    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee32aa7e-1e71-4965-b484-ef3e2aa8518e" (UID: "ee32aa7e-1e71-4965-b484-ef3e2aa8518e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:36.732831 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.732743    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-api-access-d2mdb" (OuterVolumeSpecName: "kube-api-access-d2mdb") pod "ee32aa7e-1e71-4965-b484-ef3e2aa8518e" (UID: "ee32aa7e-1e71-4965-b484-ef3e2aa8518e"). InnerVolumeSpecName "kube-api-access-d2mdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:36.739865 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.739839    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee32aa7e-1e71-4965-b484-ef3e2aa8518e" (UID: "ee32aa7e-1e71-4965-b484-ef3e2aa8518e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:36.831778 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.831748    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:18:36.831778 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.831773    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:18:36.831778 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.831783    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2mdb\" (UniqueName: \"kubernetes.io/projected/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-api-access-d2mdb\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:18:36.831981 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.831793    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:18:36.831981 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:36.831801    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee32aa7e-1e71-4965-b484-ef3e2aa8518e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:18:37.102149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:37.102070    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"
Sep 16 19:18:37.102585 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:37.102065    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6" event={"ID":"ee32aa7e-1e71-4965-b484-ef3e2aa8518e","Type":"ContainerDied","Data":"125ba776bac64482dc6e392f7c5b7576788943c80f42cc268fa97744b3fa3e72"}
Sep 16 19:18:37.102585 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:37.102223    2584 scope.go:117] "RemoveContainer" containerID="d134c76039253cabe07750990c60001e580714e6d841e72a2960f49864a69d8b"
Sep 16 19:18:37.111186 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:37.111163    2584 scope.go:117] "RemoveContainer" containerID="16016f3571d1a3b9a118e435dd6c3a3daa3cb61539505f0e1bd4e30395dc0e17"
Sep 16 19:18:37.118416 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:37.118399    2584 scope.go:117] "RemoveContainer" containerID="4db8b4134525013b3be7c5d4b4eec9b82141c5a923659cdcb1be54bebe09cb6d"
Sep 16 19:18:37.125050 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:37.125029    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"]
Sep 16 19:18:37.128035 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:37.128016    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xqjk6"]
Sep 16 19:18:37.816070 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:37.816028    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" path="/var/lib/kubelet/pods/ee32aa7e-1e71-4965-b484-ef3e2aa8518e/volumes"
Sep 16 19:18:56.581438 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581359    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"]
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581733    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container"
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581746    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container"
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581757    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy"
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581762    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy"
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581771    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kube-rbac-proxy"
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581777    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kube-rbac-proxy"
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581787    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kserve-container"
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581792    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kserve-container"
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581808    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="storage-initializer"
Sep 16 19:18:56.581808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581814    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="storage-initializer"
Sep 16 19:18:56.582146 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581830    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="storage-initializer"
Sep 16 19:18:56.582146 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581835    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="storage-initializer"
Sep 16 19:18:56.582146 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581884    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kube-rbac-proxy"
Sep 16 19:18:56.582146 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581890    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kserve-container"
Sep 16 19:18:56.582146 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581896    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee32aa7e-1e71-4965-b484-ef3e2aa8518e" containerName="kube-rbac-proxy"
Sep 16 19:18:56.582146 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.581902    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f641c9b8-93f2-4504-a371-68d9639d34c5" containerName="kserve-container"
Sep 16 19:18:56.585025 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.585007    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.587597 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.587574    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 16 19:18:56.587755 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.587580    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 16 19:18:56.589183 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.589159    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jchj7\" (UniqueName: \"kubernetes.io/projected/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-api-access-jchj7\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.589256 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.589197    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d47624-1592-4419-8c7b-fba5a250e8d9-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.589256 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.589223    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.589389 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.589333    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.589389 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.589378    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d47624-1592-4419-8c7b-fba5a250e8d9-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.596107 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.596080    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"]
Sep 16 19:18:56.689979 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.689943    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jchj7\" (UniqueName: \"kubernetes.io/projected/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-api-access-jchj7\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.690144 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.689986    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d47624-1592-4419-8c7b-fba5a250e8d9-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.690144 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.690003    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.690144 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.690114    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.690346 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.690149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d47624-1592-4419-8c7b-fba5a250e8d9-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.690346 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:18:56.690310    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 16 19:18:56.690461 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:18:56.690373    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d47624-1592-4419-8c7b-fba5a250e8d9-proxy-tls podName:77d47624-1592-4419-8c7b-fba5a250e8d9 nodeName:}" failed. No retries permitted until 2026-09-16 19:18:57.190354146 +0000 UTC m=+2067.857293278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77d47624-1592-4419-8c7b-fba5a250e8d9-proxy-tls") pod "isvc-triton-predictor-5d7db58676-q2njr" (UID: "77d47624-1592-4419-8c7b-fba5a250e8d9") : secret "isvc-triton-predictor-serving-cert" not found
Sep 16 19:18:56.690461 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.690426    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d47624-1592-4419-8c7b-fba5a250e8d9-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.690776 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.690756    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.690850 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.690780    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:56.702676 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:56.702649    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jchj7\" (UniqueName: \"kubernetes.io/projected/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-api-access-jchj7\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:57.192450 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:57.192412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d47624-1592-4419-8c7b-fba5a250e8d9-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:57.194839 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:57.194809    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d47624-1592-4419-8c7b-fba5a250e8d9-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-q2njr\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:57.196782 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:57.196757    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:18:57.473211 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:57.473187    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"]
Sep 16 19:18:57.475574 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:18:57.475542    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d47624_1592_4419_8c7b_fba5a250e8d9.slice/crio-970b93bba4709cb1ebde6ea0725ca5aec501d674442aada4bd8db37f27b2e350 WatchSource:0}: Error finding container 970b93bba4709cb1ebde6ea0725ca5aec501d674442aada4bd8db37f27b2e350: Status 404 returned error can't find the container with id 970b93bba4709cb1ebde6ea0725ca5aec501d674442aada4bd8db37f27b2e350
Sep 16 19:18:58.206718 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:58.206679    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" event={"ID":"77d47624-1592-4419-8c7b-fba5a250e8d9","Type":"ContainerStarted","Data":"9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa"}
Sep 16 19:18:58.207227 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:18:58.206749    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" event={"ID":"77d47624-1592-4419-8c7b-fba5a250e8d9","Type":"ContainerStarted","Data":"970b93bba4709cb1ebde6ea0725ca5aec501d674442aada4bd8db37f27b2e350"}
Sep 16 19:19:02.226970 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:02.226938    2584 generic.go:356] "Generic (PLEG): container finished" podID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerID="9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa" exitCode=0
Sep 16 19:19:02.227372 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:02.227010    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" event={"ID":"77d47624-1592-4419-8c7b-fba5a250e8d9","Type":"ContainerDied","Data":"9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa"}
Sep 16 19:19:25.167116 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:25.167033    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"]
Sep 16 19:19:25.167681 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:25.167339    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" containerID="cri-o://e2e95c084e512391084468d1c381ff4be254d1beef07ec5fb79a4f3b284f7472" gracePeriod=30
Sep 16 19:19:25.167681 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:25.167503    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kube-rbac-proxy" containerID="cri-o://0feaea79b66fbafadb440f52d39594414021494d28236c87b5c1291cb583efd7" gracePeriod=30
Sep 16 19:19:25.409056 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:25.409016    2584 generic.go:356] "Generic (PLEG): container finished" podID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerID="0feaea79b66fbafadb440f52d39594414021494d28236c87b5c1291cb583efd7" exitCode=2
Sep 16 19:19:25.409366 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:25.409125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" event={"ID":"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e","Type":"ContainerDied","Data":"0feaea79b66fbafadb440f52d39594414021494d28236c87b5c1291cb583efd7"}
Sep 16 19:19:26.811918 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:26.811513    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 16 19:19:27.498307 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:27.498194    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 16 19:19:30.450863 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.450824    2584 generic.go:356] "Generic (PLEG): container finished" podID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerID="e2e95c084e512391084468d1c381ff4be254d1beef07ec5fb79a4f3b284f7472" exitCode=0
Sep 16 19:19:30.451329 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.450921    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" event={"ID":"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e","Type":"ContainerDied","Data":"e2e95c084e512391084468d1c381ff4be254d1beef07ec5fb79a4f3b284f7472"}
Sep 16 19:19:30.531923 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.531891    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"]
Sep 16 19:19:30.556928 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.556822    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"]
Sep 16 19:19:30.557239 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.557144    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.561527 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.561017    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:19:30.562632 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.562364    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:19:30.626852 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.626819    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:19:30.690945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.690913    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kserve-provision-location\") pod \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") "
Sep 16 19:19:30.691100 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.690965    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6rbz\" (UniqueName: \"kubernetes.io/projected/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-api-access-p6rbz\") pod \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") "
Sep 16 19:19:30.691100 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691042    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-rbac-proxy-sar-config\") pod \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") "
Sep 16 19:19:30.691304 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691250    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" (UID: "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:30.691407 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691308    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-cabundle-cert\") pod \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") "
Sep 16 19:19:30.691407 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691371    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-proxy-tls\") pod \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\" (UID: \"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e\") "
Sep 16 19:19:30.691682 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691449    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" (UID: "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:30.691682 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691605    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.691812 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691656    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5ht\" (UniqueName: \"kubernetes.io/projected/14bd9123-b917-431d-876b-3fe8458a0086-kube-api-access-rw5ht\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.691812 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691734    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" (UID: "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:30.691812 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691754    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.692000 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691873    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bd9123-b917-431d-876b-3fe8458a0086-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.692000 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bd9123-b917-431d-876b-3fe8458a0086-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.692136 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.691994    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:19:30.692136 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.692029    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:19:30.692136 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.692046    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:19:30.694000 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.693975    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" (UID: "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:30.695053 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.695024    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-api-access-p6rbz" (OuterVolumeSpecName: "kube-api-access-p6rbz") pod "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" (UID: "1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e"). InnerVolumeSpecName "kube-api-access-p6rbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:30.793332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.792951    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.793332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.793014    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5ht\" (UniqueName: \"kubernetes.io/projected/14bd9123-b917-431d-876b-3fe8458a0086-kube-api-access-rw5ht\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.793332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.793060    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.793332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.793115    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bd9123-b917-431d-876b-3fe8458a0086-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.793332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.793141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bd9123-b917-431d-876b-3fe8458a0086-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.793332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.793197    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:19:30.793332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.793214    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6rbz\" (UniqueName: \"kubernetes.io/projected/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e-kube-api-access-p6rbz\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:19:30.793893 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.793579    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bd9123-b917-431d-876b-3fe8458a0086-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.793893 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:19:30.793693    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:19:30.793893 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:19:30.793751    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14bd9123-b917-431d-876b-3fe8458a0086-proxy-tls podName:14bd9123-b917-431d-876b-3fe8458a0086 nodeName:}" failed. No retries permitted until 2026-09-16 19:19:31.293731346 +0000 UTC m=+2101.960670482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/14bd9123-b917-431d-876b-3fe8458a0086-proxy-tls") pod "isvc-canary-rollback-predictor-857b7ffb9f-kf56n" (UID: "14bd9123-b917-431d-876b-3fe8458a0086") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:19:30.793893 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.793781    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.794188 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.794093    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:30.803958 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:30.803931    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5ht\" (UniqueName: \"kubernetes.io/projected/14bd9123-b917-431d-876b-3fe8458a0086-kube-api-access-rw5ht\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:31.297901 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.297865    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bd9123-b917-431d-876b-3fe8458a0086-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:31.300879 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.300847    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bd9123-b917-431d-876b-3fe8458a0086-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-kf56n\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:31.459778 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.459737    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml" event={"ID":"1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e","Type":"ContainerDied","Data":"ffbf8e1a18e6c4476d3dafb2dfd95f8587ab9cb2281441ebac2ef55bb7bca66c"}
Sep 16 19:19:31.459778 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.459765    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"
Sep 16 19:19:31.460282 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.459827    2584 scope.go:117] "RemoveContainer" containerID="0feaea79b66fbafadb440f52d39594414021494d28236c87b5c1291cb583efd7"
Sep 16 19:19:31.474014 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.473995    2584 scope.go:117] "RemoveContainer" containerID="e2e95c084e512391084468d1c381ff4be254d1beef07ec5fb79a4f3b284f7472"
Sep 16 19:19:31.483911 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.483891    2584 scope.go:117] "RemoveContainer" containerID="d8e0735f09ad0b4bd07c55d4a393dbe97e520e68245091bd977cc525e61e1ac1"
Sep 16 19:19:31.489924 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.489895    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"]
Sep 16 19:19:31.494474 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.494452    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-2r4ml"]
Sep 16 19:19:31.494589 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.494469    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:31.819852 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.819643    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" path="/var/lib/kubelet/pods/1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e/volumes"
Sep 16 19:19:31.822617 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:31.822546    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"]
Sep 16 19:19:31.825101 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:19:31.825070    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14bd9123_b917_431d_876b_3fe8458a0086.slice/crio-ee481a91d2c2d76a0cca68da1e76a19a195c3ac2ec9b3679cd3b1b8b05e8c5a7 WatchSource:0}: Error finding container ee481a91d2c2d76a0cca68da1e76a19a195c3ac2ec9b3679cd3b1b8b05e8c5a7: Status 404 returned error can't find the container with id ee481a91d2c2d76a0cca68da1e76a19a195c3ac2ec9b3679cd3b1b8b05e8c5a7
Sep 16 19:19:32.470911 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:32.470757    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" event={"ID":"14bd9123-b917-431d-876b-3fe8458a0086","Type":"ContainerStarted","Data":"00b601dcc5cbdd650b1161df48e6fd8dc19956d670ecd2b46bbe3b61cf249524"}
Sep 16 19:19:32.470911 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:32.470824    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" event={"ID":"14bd9123-b917-431d-876b-3fe8458a0086","Type":"ContainerStarted","Data":"ee481a91d2c2d76a0cca68da1e76a19a195c3ac2ec9b3679cd3b1b8b05e8c5a7"}
Sep 16 19:19:36.499651 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:36.499609    2584 generic.go:356] "Generic (PLEG): container finished" podID="14bd9123-b917-431d-876b-3fe8458a0086" containerID="00b601dcc5cbdd650b1161df48e6fd8dc19956d670ecd2b46bbe3b61cf249524" exitCode=0
Sep 16 19:19:36.500219 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:36.499682    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" event={"ID":"14bd9123-b917-431d-876b-3fe8458a0086","Type":"ContainerDied","Data":"00b601dcc5cbdd650b1161df48e6fd8dc19956d670ecd2b46bbe3b61cf249524"}
Sep 16 19:19:37.508987 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:37.508932    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" event={"ID":"14bd9123-b917-431d-876b-3fe8458a0086","Type":"ContainerStarted","Data":"e200631953162b8c088a77bbe9a83d8faeebfa354a33fdf7e251ec59d7f936ed"}
Sep 16 19:19:37.509713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:37.509011    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" event={"ID":"14bd9123-b917-431d-876b-3fe8458a0086","Type":"ContainerStarted","Data":"b1525a5ea1f198331b24b4f88fffc53d5cb57f1364f6e303be5d29415c2136ad"}
Sep 16 19:19:37.509713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:37.509677    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:37.509898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:37.509739    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:37.511437 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:37.511290    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:37.532997 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:37.532868    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podStartSLOduration=3.418764899 podStartE2EDuration="7.532855999s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.1140911s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:37.532030875 +0000 UTC m=+2108.198970051" watchObservedRunningTime="2026-09-16 19:19:37.532855999 +0000 UTC m=+2108.199795145"
Sep 16 19:19:38.515410 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:38.515330    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:43.521854 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:43.521512    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:19:43.522438 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:43.522120    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:19:53.522664 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:19:53.522622    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:03.522951 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:03.522904    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:13.522980 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:13.522941    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:23.523052 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:23.522959    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:33.522244 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:33.522199    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:43.522610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:43.522564    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:53.522069 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:53.522024    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:53.777673 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:53.777574    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"]
Sep 16 19:20:53.777927 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:53.777897    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" containerID="cri-o://b1525a5ea1f198331b24b4f88fffc53d5cb57f1364f6e303be5d29415c2136ad" gracePeriod=30
Sep 16 19:20:53.778491 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:53.778035    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy" containerID="cri-o://e200631953162b8c088a77bbe9a83d8faeebfa354a33fdf7e251ec59d7f936ed" gracePeriod=30
Sep 16 19:20:53.778491 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:53.778187    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:20:54.929810 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:54.929768    2584 generic.go:356] "Generic (PLEG): container finished" podID="14bd9123-b917-431d-876b-3fe8458a0086" containerID="e200631953162b8c088a77bbe9a83d8faeebfa354a33fdf7e251ec59d7f936ed" exitCode=2
Sep 16 19:20:54.930203 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:54.929841    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" event={"ID":"14bd9123-b917-431d-876b-3fe8458a0086","Type":"ContainerDied","Data":"e200631953162b8c088a77bbe9a83d8faeebfa354a33fdf7e251ec59d7f936ed"}
Sep 16 19:20:56.941510 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:56.941473    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" event={"ID":"77d47624-1592-4419-8c7b-fba5a250e8d9","Type":"ContainerStarted","Data":"f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0"}
Sep 16 19:20:56.942006 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:56.941535    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" event={"ID":"77d47624-1592-4419-8c7b-fba5a250e8d9","Type":"ContainerStarted","Data":"e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8"}
Sep 16 19:20:56.942006 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:56.941601    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:20:56.967187 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:56.967131    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" podStartSLOduration=2.795584548 podStartE2EDuration="2m0.967113696s" totalImagesPullingTime="1m54.116279935s" totalInitContainerRuntime="4.055249213s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:56.965231448 +0000 UTC m=+2187.632170607" watchObservedRunningTime="2026-09-16 19:20:56.967113696 +0000 UTC m=+2187.634052842"
Sep 16 19:20:57.946855 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:57.946816    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:20:57.948157 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:57.948130    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 16 19:20:58.516539 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:58.516490    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 16 19:20:58.951094 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:20:58.951054    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 16 19:21:00.527371 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.527332    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"]
Sep 16 19:21:00.527882 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.527862    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="storage-initializer"
Sep 16 19:21:00.527965 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.527885    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="storage-initializer"
Sep 16 19:21:00.527965 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.527926    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container"
Sep 16 19:21:00.527965 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.527936    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container"
Sep 16 19:21:00.527965 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.527947    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kube-rbac-proxy"
Sep 16 19:21:00.527965 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.527955    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kube-rbac-proxy"
Sep 16 19:21:00.528220 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.528049    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kube-rbac-proxy"
Sep 16 19:21:00.528220 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.528064    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1840b2e4-9d55-4bbb-a1c8-1ec3a35aba5e" containerName="kserve-container"
Sep 16 19:21:00.549093 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.549056    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"]
Sep 16 19:21:00.549308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.549198    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.593731 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.593687    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.593731 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.593733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.593944 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.593756    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.593944 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.593803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r267j\" (UniqueName: \"kubernetes.io/projected/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-api-access-r267j\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.593944 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.593896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.694415 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.694368    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.694605 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.694433    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.694605 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.694453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.694605 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.694467    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r267j\" (UniqueName: \"kubernetes.io/projected/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-api-access-r267j\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.694605 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.694498    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.695056 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.695031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.695259 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.695239    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.695346 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.695329    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.696947 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.696931    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.704451 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.704425    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r267j\" (UniqueName: \"kubernetes.io/projected/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-api-access-r267j\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-5xnmm\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:00.860858 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:00.860763    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:01.255835 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:01.255805    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"]
Sep 16 19:21:01.257578 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:21:01.257539    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c85e8b_51b6_4cba_af27_0c0e4b84c8f7.slice/crio-5a702d8469663531333a76d911987b38d07651b36cc020f70426b9d271a1d0ad WatchSource:0}: Error finding container 5a702d8469663531333a76d911987b38d07651b36cc020f70426b9d271a1d0ad: Status 404 returned error can't find the container with id 5a702d8469663531333a76d911987b38d07651b36cc020f70426b9d271a1d0ad
Sep 16 19:21:01.967136 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:01.967099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" event={"ID":"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7","Type":"ContainerStarted","Data":"bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c"}
Sep 16 19:21:01.967721 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:01.967170    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" event={"ID":"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7","Type":"ContainerStarted","Data":"5a702d8469663531333a76d911987b38d07651b36cc020f70426b9d271a1d0ad"}
Sep 16 19:21:03.516192 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:03.516149    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 16 19:21:03.779290 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:03.779177    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 16 19:21:03.955745 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:03.955711    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:21:03.956540 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:03.956516    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:21:05.987163 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:05.987130    2584 generic.go:356] "Generic (PLEG): container finished" podID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerID="bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c" exitCode=0
Sep 16 19:21:05.987564 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:05.987201    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" event={"ID":"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7","Type":"ContainerDied","Data":"bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c"}
Sep 16 19:21:06.994921 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:06.994886    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" event={"ID":"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7","Type":"ContainerStarted","Data":"05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63"}
Sep 16 19:21:06.995355 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:06.994952    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" event={"ID":"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7","Type":"ContainerStarted","Data":"551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb"}
Sep 16 19:21:06.995355 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:06.995140    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:07.018719 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:07.018650    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podStartSLOduration=2.537461779 podStartE2EDuration="7.018627073s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.481165294s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:07.017396281 +0000 UTC m=+2197.684335454" watchObservedRunningTime="2026-09-16 19:21:07.018627073 +0000 UTC m=+2197.685566220"
Sep 16 19:21:08.000311 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:08.000246    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:08.001833 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:08.001802    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:21:08.516071 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:08.516022    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 16 19:21:08.516250 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:08.516155    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:21:08.677417 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:08.677382    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"]
Sep 16 19:21:08.677653 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:08.677631    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kserve-container" containerID="cri-o://e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8" gracePeriod=30
Sep 16 19:21:08.677727 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:08.677701    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kube-rbac-proxy" containerID="cri-o://f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0" gracePeriod=30
Sep 16 19:21:08.952180 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:08.952133    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 16 19:21:09.007337 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:09.007301    2584 generic.go:356] "Generic (PLEG): container finished" podID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerID="f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0" exitCode=2
Sep 16 19:21:09.007816 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:09.007372    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" event={"ID":"77d47624-1592-4419-8c7b-fba5a250e8d9","Type":"ContainerDied","Data":"f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0"}
Sep 16 19:21:09.007816 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:09.007639    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:21:11.770595 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.770514    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:21:11.898037 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.898000    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-rbac-proxy-sar-config\") pod \"77d47624-1592-4419-8c7b-fba5a250e8d9\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") "
Sep 16 19:21:11.898223 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.898103    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-cabundle-cert\") pod \"77d47624-1592-4419-8c7b-fba5a250e8d9\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") "
Sep 16 19:21:11.898223 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.898147    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jchj7\" (UniqueName: \"kubernetes.io/projected/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-api-access-jchj7\") pod \"77d47624-1592-4419-8c7b-fba5a250e8d9\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") "
Sep 16 19:21:11.898223 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.898164    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d47624-1592-4419-8c7b-fba5a250e8d9-kserve-provision-location\") pod \"77d47624-1592-4419-8c7b-fba5a250e8d9\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") "
Sep 16 19:21:11.898223 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.898187    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d47624-1592-4419-8c7b-fba5a250e8d9-proxy-tls\") pod \"77d47624-1592-4419-8c7b-fba5a250e8d9\" (UID: \"77d47624-1592-4419-8c7b-fba5a250e8d9\") "
Sep 16 19:21:11.898537 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.898498    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77d47624-1592-4419-8c7b-fba5a250e8d9" (UID: "77d47624-1592-4419-8c7b-fba5a250e8d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:11.898537 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.898514    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77d47624-1592-4419-8c7b-fba5a250e8d9" (UID: "77d47624-1592-4419-8c7b-fba5a250e8d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:11.898705 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.898622    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d47624-1592-4419-8c7b-fba5a250e8d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77d47624-1592-4419-8c7b-fba5a250e8d9" (UID: "77d47624-1592-4419-8c7b-fba5a250e8d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:11.900464 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.900436    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-api-access-jchj7" (OuterVolumeSpecName: "kube-api-access-jchj7") pod "77d47624-1592-4419-8c7b-fba5a250e8d9" (UID: "77d47624-1592-4419-8c7b-fba5a250e8d9"). InnerVolumeSpecName "kube-api-access-jchj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:11.900571 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.900493    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d47624-1592-4419-8c7b-fba5a250e8d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77d47624-1592-4419-8c7b-fba5a250e8d9" (UID: "77d47624-1592-4419-8c7b-fba5a250e8d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:11.999682 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.999644    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jchj7\" (UniqueName: \"kubernetes.io/projected/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-api-access-jchj7\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:11.999682 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.999678    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d47624-1592-4419-8c7b-fba5a250e8d9-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:11.999682 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.999688    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d47624-1592-4419-8c7b-fba5a250e8d9-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:11.999939 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.999698    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:11.999939 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:11.999708    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d47624-1592-4419-8c7b-fba5a250e8d9-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:12.024116 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.024026    2584 generic.go:356] "Generic (PLEG): container finished" podID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerID="e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8" exitCode=0
Sep 16 19:21:12.024116 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.024084    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"
Sep 16 19:21:12.024336 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.024111    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" event={"ID":"77d47624-1592-4419-8c7b-fba5a250e8d9","Type":"ContainerDied","Data":"e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8"}
Sep 16 19:21:12.024336 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.024173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr" event={"ID":"77d47624-1592-4419-8c7b-fba5a250e8d9","Type":"ContainerDied","Data":"970b93bba4709cb1ebde6ea0725ca5aec501d674442aada4bd8db37f27b2e350"}
Sep 16 19:21:12.024336 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.024232    2584 scope.go:117] "RemoveContainer" containerID="f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0"
Sep 16 19:21:12.033843 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.033821    2584 scope.go:117] "RemoveContainer" containerID="e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8"
Sep 16 19:21:12.041962 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.041942    2584 scope.go:117] "RemoveContainer" containerID="9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa"
Sep 16 19:21:12.049813 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.049792    2584 scope.go:117] "RemoveContainer" containerID="f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0"
Sep 16 19:21:12.050115 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:21:12.050096    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0\": container with ID starting with f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0 not found: ID does not exist" containerID="f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0"
Sep 16 19:21:12.050188 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.050123    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0"} err="failed to get container status \"f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0\": rpc error: code = NotFound desc = could not find container \"f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0\": container with ID starting with f7302a328233a700a1bc42b8286979a9f274e92a118ad88d4e89860324b0b9e0 not found: ID does not exist"
Sep 16 19:21:12.050188 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.050140    2584 scope.go:117] "RemoveContainer" containerID="e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8"
Sep 16 19:21:12.050408 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:21:12.050392    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8\": container with ID starting with e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8 not found: ID does not exist" containerID="e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8"
Sep 16 19:21:12.050459 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.050412    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8"} err="failed to get container status \"e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8\": rpc error: code = NotFound desc = could not find container \"e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8\": container with ID starting with e5923d204791f7ca5f538c26ae62aaba026698a06015f92cfe0d454ca2adf3d8 not found: ID does not exist"
Sep 16 19:21:12.050459 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.050425    2584 scope.go:117] "RemoveContainer" containerID="9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa"
Sep 16 19:21:12.050681 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:21:12.050664    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa\": container with ID starting with 9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa not found: ID does not exist" containerID="9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa"
Sep 16 19:21:12.050731 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.050684    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa"} err="failed to get container status \"9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa\": rpc error: code = NotFound desc = could not find container \"9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa\": container with ID starting with 9185a1059b49541c030a82d3238edc4e88e88fe4717df5014c47440b7a13fbfa not found: ID does not exist"
Sep 16 19:21:12.056451 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.056425    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"]
Sep 16 19:21:12.065233 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:12.065206    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-q2njr"]
Sep 16 19:21:13.515730 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:13.515689    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 16 19:21:13.779989 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:13.779908    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:21:13.819356 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:13.819288    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" path="/var/lib/kubelet/pods/77d47624-1592-4419-8c7b-fba5a250e8d9/volumes"
Sep 16 19:21:14.014001 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:14.013971    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:21:14.014531 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:14.014496    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:21:18.515694 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:18.515653    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 16 19:21:23.515748 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:23.515709    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 16 19:21:24.014611 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.014574    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:21:24.087706 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.087674    2584 generic.go:356] "Generic (PLEG): container finished" podID="14bd9123-b917-431d-876b-3fe8458a0086" containerID="b1525a5ea1f198331b24b4f88fffc53d5cb57f1364f6e303be5d29415c2136ad" exitCode=137
Sep 16 19:21:24.087856 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.087715    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" event={"ID":"14bd9123-b917-431d-876b-3fe8458a0086","Type":"ContainerDied","Data":"b1525a5ea1f198331b24b4f88fffc53d5cb57f1364f6e303be5d29415c2136ad"}
Sep 16 19:21:24.106396 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.106365    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:21:24.205202 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.205161    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-cabundle-cert\") pod \"14bd9123-b917-431d-876b-3fe8458a0086\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") "
Sep 16 19:21:24.205420 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.205222    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bd9123-b917-431d-876b-3fe8458a0086-proxy-tls\") pod \"14bd9123-b917-431d-876b-3fe8458a0086\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") "
Sep 16 19:21:24.205420 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.205288    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw5ht\" (UniqueName: \"kubernetes.io/projected/14bd9123-b917-431d-876b-3fe8458a0086-kube-api-access-rw5ht\") pod \"14bd9123-b917-431d-876b-3fe8458a0086\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") "
Sep 16 19:21:24.205420 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.205339    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-kube-rbac-proxy-sar-config\") pod \"14bd9123-b917-431d-876b-3fe8458a0086\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") "
Sep 16 19:21:24.205420 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.205370    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bd9123-b917-431d-876b-3fe8458a0086-kserve-provision-location\") pod \"14bd9123-b917-431d-876b-3fe8458a0086\" (UID: \"14bd9123-b917-431d-876b-3fe8458a0086\") "
Sep 16 19:21:24.205651 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.205590    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14bd9123-b917-431d-876b-3fe8458a0086" (UID: "14bd9123-b917-431d-876b-3fe8458a0086"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:24.205847 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.205819    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14bd9123-b917-431d-876b-3fe8458a0086-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14bd9123-b917-431d-876b-3fe8458a0086" (UID: "14bd9123-b917-431d-876b-3fe8458a0086"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:24.205925 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.205844    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14bd9123-b917-431d-876b-3fe8458a0086" (UID: "14bd9123-b917-431d-876b-3fe8458a0086"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:24.207590 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.207561    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bd9123-b917-431d-876b-3fe8458a0086-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14bd9123-b917-431d-876b-3fe8458a0086" (UID: "14bd9123-b917-431d-876b-3fe8458a0086"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:24.207590 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.207578    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14bd9123-b917-431d-876b-3fe8458a0086-kube-api-access-rw5ht" (OuterVolumeSpecName: "kube-api-access-rw5ht") pod "14bd9123-b917-431d-876b-3fe8458a0086" (UID: "14bd9123-b917-431d-876b-3fe8458a0086"). InnerVolumeSpecName "kube-api-access-rw5ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:24.307006 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.306960    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:24.307006 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.306997    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14bd9123-b917-431d-876b-3fe8458a0086-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:24.307006 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.307007    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw5ht\" (UniqueName: \"kubernetes.io/projected/14bd9123-b917-431d-876b-3fe8458a0086-kube-api-access-rw5ht\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:24.307006 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.307016    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14bd9123-b917-431d-876b-3fe8458a0086-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:24.307309 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:24.307025    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14bd9123-b917-431d-876b-3fe8458a0086-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.094026 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:25.093997    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"
Sep 16 19:21:25.094026 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:25.094005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n" event={"ID":"14bd9123-b917-431d-876b-3fe8458a0086","Type":"ContainerDied","Data":"ee481a91d2c2d76a0cca68da1e76a19a195c3ac2ec9b3679cd3b1b8b05e8c5a7"}
Sep 16 19:21:25.094526 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:25.094081    2584 scope.go:117] "RemoveContainer" containerID="e200631953162b8c088a77bbe9a83d8faeebfa354a33fdf7e251ec59d7f936ed"
Sep 16 19:21:25.102983 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:25.102964    2584 scope.go:117] "RemoveContainer" containerID="b1525a5ea1f198331b24b4f88fffc53d5cb57f1364f6e303be5d29415c2136ad"
Sep 16 19:21:25.112759 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:25.112736    2584 scope.go:117] "RemoveContainer" containerID="00b601dcc5cbdd650b1161df48e6fd8dc19956d670ecd2b46bbe3b61cf249524"
Sep 16 19:21:25.125774 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:25.125742    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"]
Sep 16 19:21:25.128801 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:25.128770    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-kf56n"]
Sep 16 19:21:25.816622 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:25.816584    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14bd9123-b917-431d-876b-3fe8458a0086" path="/var/lib/kubelet/pods/14bd9123-b917-431d-876b-3fe8458a0086/volumes"
Sep 16 19:21:34.015406 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:34.015353    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:21:44.014606 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:44.014560    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:21:54.015577 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:21:54.015482    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:22:04.014426 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:04.014385    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:22:14.015325 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:14.015283    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:22:24.015461 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:24.015424    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:22:30.544580 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:30.544549    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"]
Sep 16 19:22:30.547019 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:30.544834    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" containerID="cri-o://551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb" gracePeriod=30
Sep 16 19:22:30.547019 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:30.544843    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kube-rbac-proxy" containerID="cri-o://05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63" gracePeriod=30
Sep 16 19:22:31.407477 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:31.407436    2584 generic.go:356] "Generic (PLEG): container finished" podID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerID="05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63" exitCode=2
Sep 16 19:22:31.407646 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:31.407493    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" event={"ID":"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7","Type":"ContainerDied","Data":"05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63"}
Sep 16 19:22:34.008742 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:34.008697    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 16 19:22:34.014965 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:34.014938    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 16 19:22:35.084197 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.084174    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:22:35.098850 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.098830    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r267j\" (UniqueName: \"kubernetes.io/projected/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-api-access-r267j\") pod \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") "
Sep 16 19:22:35.098942 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.098863    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-proxy-tls\") pod \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") "
Sep 16 19:22:35.098942 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.098887    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kserve-provision-location\") pod \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") "
Sep 16 19:22:35.098942 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.098909    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-cabundle-cert\") pod \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") "
Sep 16 19:22:35.099107 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.098961    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-rbac-proxy-sar-config\") pod \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\" (UID: \"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7\") "
Sep 16 19:22:35.099322 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.099289    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" (UID: "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:35.099431 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.099333    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" (UID: "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:35.099431 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.099342    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" (UID: "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:35.100907 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.100886    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" (UID: "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:35.101006 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.100890    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-api-access-r267j" (OuterVolumeSpecName: "kube-api-access-r267j") pod "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" (UID: "e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7"). InnerVolumeSpecName "kube-api-access-r267j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:35.199943 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.199914    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.199943 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.199944    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.200135 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.199954    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r267j\" (UniqueName: \"kubernetes.io/projected/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kube-api-access-r267j\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.200135 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.199963    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.200135 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.199972    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.427203 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.427168    2584 generic.go:356] "Generic (PLEG): container finished" podID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerID="551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb" exitCode=0
Sep 16 19:22:35.427383 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.427224    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"
Sep 16 19:22:35.427383 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.427241    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" event={"ID":"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7","Type":"ContainerDied","Data":"551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb"}
Sep 16 19:22:35.427383 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.427314    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm" event={"ID":"e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7","Type":"ContainerDied","Data":"5a702d8469663531333a76d911987b38d07651b36cc020f70426b9d271a1d0ad"}
Sep 16 19:22:35.427383 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.427342    2584 scope.go:117] "RemoveContainer" containerID="05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63"
Sep 16 19:22:35.436407 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.436390    2584 scope.go:117] "RemoveContainer" containerID="551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb"
Sep 16 19:22:35.443989 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.443970    2584 scope.go:117] "RemoveContainer" containerID="bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c"
Sep 16 19:22:35.451082 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.451067    2584 scope.go:117] "RemoveContainer" containerID="05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63"
Sep 16 19:22:35.451358 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:22:35.451340    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63\": container with ID starting with 05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63 not found: ID does not exist" containerID="05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63"
Sep 16 19:22:35.451413 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.451367    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63"} err="failed to get container status \"05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63\": rpc error: code = NotFound desc = could not find container \"05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63\": container with ID starting with 05ebb637297bf775174d6083f9adb56f3bc0363276326bc4b912a60509663a63 not found: ID does not exist"
Sep 16 19:22:35.451413 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.451390    2584 scope.go:117] "RemoveContainer" containerID="551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb"
Sep 16 19:22:35.451692 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:22:35.451671    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb\": container with ID starting with 551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb not found: ID does not exist" containerID="551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb"
Sep 16 19:22:35.451747 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.451701    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb"} err="failed to get container status \"551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb\": rpc error: code = NotFound desc = could not find container \"551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb\": container with ID starting with 551e2dc633cbeea40f0980d5cb58f91b75b0730f6727371c8d87e7db9e178bbb not found: ID does not exist"
Sep 16 19:22:35.451747 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.451717    2584 scope.go:117] "RemoveContainer" containerID="bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c"
Sep 16 19:22:35.451991 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:22:35.451975    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c\": container with ID starting with bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c not found: ID does not exist" containerID="bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c"
Sep 16 19:22:35.452044 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.451996    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c"} err="failed to get container status \"bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c\": rpc error: code = NotFound desc = could not find container \"bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c\": container with ID starting with bd98a87215ad445cf6eff8bb7e17eb68db441fb7bf6494a2843600ad36b1967c not found: ID does not exist"
Sep 16 19:22:35.454992 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.454971    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"]
Sep 16 19:22:35.464843 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.464822    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-5xnmm"]
Sep 16 19:22:35.816610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.816535    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" path="/var/lib/kubelet/pods/e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7/volumes"
Sep 16 19:22:35.941446 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941412    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"]
Sep 16 19:22:35.941737 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941725    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="storage-initializer"
Sep 16 19:22:35.941792 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941738    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="storage-initializer"
Sep 16 19:22:35.941792 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941749    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="storage-initializer"
Sep 16 19:22:35.941792 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941755    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="storage-initializer"
Sep 16 19:22:35.941792 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941781    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kserve-container"
Sep 16 19:22:35.941792 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941787    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kserve-container"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941794    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941800    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941806    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kube-rbac-proxy"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941811    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kube-rbac-proxy"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941817    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kube-rbac-proxy"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941821    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kube-rbac-proxy"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941829    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="storage-initializer"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941834    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="storage-initializer"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941839    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941844    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941850    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941855    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941899    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kserve-container"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941907    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="14bd9123-b917-431d-876b-3fe8458a0086" containerName="kube-rbac-proxy"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941912    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kserve-container"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941916    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d47624-1592-4419-8c7b-fba5a250e8d9" containerName="kube-rbac-proxy"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941922    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kube-rbac-proxy"
Sep 16 19:22:35.941945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.941929    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c85e8b-51b6-4cba-af27-0c0e4b84c8f7" containerName="kserve-container"
Sep 16 19:22:35.945150 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.945132    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:35.956543 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.956520    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:22:35.956662 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.956557    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:22:35.956662 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.956627    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:22:35.957097 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.957083    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 19:22:35.957729 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.957714    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:22:35.957729 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.957723    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:22:35.957882 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.957723    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:22:35.996165 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:35.996137    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"]
Sep 16 19:22:36.006541 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.006519    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.006677 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.006568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gc4j\" (UniqueName: \"kubernetes.io/projected/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-api-access-8gc4j\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.006677 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.006588    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.006677 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.006652    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.006852 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.006690    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.107624 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.107537    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.107624 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.107581    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.108164 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.107636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.108164 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.107696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gc4j\" (UniqueName: \"kubernetes.io/projected/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-api-access-8gc4j\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.108164 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.107725    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.108164 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.108052    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.108343 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.108240    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.108343 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.108318    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.110242 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.110217    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.123576 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.123556    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"]
Sep 16 19:22:36.127192 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.127176    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.131399 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.131382    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:22:36.137840 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.137820    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gc4j\" (UniqueName: \"kubernetes.io/projected/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-api-access-8gc4j\") pod \"isvc-canary-stop-v2-predictor-84ff76f474-zv7d7\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.146224 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.146202    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"]
Sep 16 19:22:36.208956 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.208926    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ad6e3e-f147-453d-9609-0f44347ce117-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.209111 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.208972    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.209111 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.208995    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.209111 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.209069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ad6e3e-f147-453d-9609-0f44347ce117-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.209222 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.209109    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdlp9\" (UniqueName: \"kubernetes.io/projected/43ad6e3e-f147-453d-9609-0f44347ce117-kube-api-access-cdlp9\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.255097 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.255068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:36.310287 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.310237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ad6e3e-f147-453d-9609-0f44347ce117-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.310415 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.310347    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.310415 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.310377    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.310415 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.310408    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ad6e3e-f147-453d-9609-0f44347ce117-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.310591 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.310440    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdlp9\" (UniqueName: \"kubernetes.io/projected/43ad6e3e-f147-453d-9609-0f44347ce117-kube-api-access-cdlp9\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.313724 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.310673    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ad6e3e-f147-453d-9609-0f44347ce117-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.313724 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.311349    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.313724 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.313684    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.316213 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.316173    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ad6e3e-f147-453d-9609-0f44347ce117-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.320755 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.320732    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdlp9\" (UniqueName: \"kubernetes.io/projected/43ad6e3e-f147-453d-9609-0f44347ce117-kube-api-access-cdlp9\") pod \"isvc-canary-stop-predictor-57d5c586ff-97qfh\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.437407 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.437376    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:36.535118 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.534962    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"]
Sep 16 19:22:36.537881 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:22:36.537846    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda682bdb3_a58f_4d64_86e2_0f8cf1993a37.slice/crio-8ba9aeca8c4e92f671ec8bfa1c60eb17937b21bd0e6f67a440880bea36e4f58d WatchSource:0}: Error finding container 8ba9aeca8c4e92f671ec8bfa1c60eb17937b21bd0e6f67a440880bea36e4f58d: Status 404 returned error can't find the container with id 8ba9aeca8c4e92f671ec8bfa1c60eb17937b21bd0e6f67a440880bea36e4f58d
Sep 16 19:22:36.742987 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:36.742959    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"]
Sep 16 19:22:36.744623 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:22:36.744596    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ad6e3e_f147_453d_9609_0f44347ce117.slice/crio-36c871efafdffc7b6a134689969d9c0c972e3a8f782c5529c8c50e5266d0f298 WatchSource:0}: Error finding container 36c871efafdffc7b6a134689969d9c0c972e3a8f782c5529c8c50e5266d0f298: Status 404 returned error can't find the container with id 36c871efafdffc7b6a134689969d9c0c972e3a8f782c5529c8c50e5266d0f298
Sep 16 19:22:37.439086 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:37.439050    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" event={"ID":"a682bdb3-a58f-4d64-86e2-0f8cf1993a37","Type":"ContainerStarted","Data":"a467adaa49560db98149eff1aecc16e511403495c5d4799e9f436d0a648075da"}
Sep 16 19:22:37.439558 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:37.439126    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" event={"ID":"a682bdb3-a58f-4d64-86e2-0f8cf1993a37","Type":"ContainerStarted","Data":"8ba9aeca8c4e92f671ec8bfa1c60eb17937b21bd0e6f67a440880bea36e4f58d"}
Sep 16 19:22:37.440684 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:37.440660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" event={"ID":"43ad6e3e-f147-453d-9609-0f44347ce117","Type":"ContainerStarted","Data":"2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623"}
Sep 16 19:22:37.440787 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:37.440708    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" event={"ID":"43ad6e3e-f147-453d-9609-0f44347ce117","Type":"ContainerStarted","Data":"36c871efafdffc7b6a134689969d9c0c972e3a8f782c5529c8c50e5266d0f298"}
Sep 16 19:22:40.455877 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:40.455845    2584 generic.go:356] "Generic (PLEG): container finished" podID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerID="a467adaa49560db98149eff1aecc16e511403495c5d4799e9f436d0a648075da" exitCode=0
Sep 16 19:22:40.456260 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:40.455920    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" event={"ID":"a682bdb3-a58f-4d64-86e2-0f8cf1993a37","Type":"ContainerDied","Data":"a467adaa49560db98149eff1aecc16e511403495c5d4799e9f436d0a648075da"}
Sep 16 19:22:41.462189 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:41.462156    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" event={"ID":"a682bdb3-a58f-4d64-86e2-0f8cf1993a37","Type":"ContainerStarted","Data":"0e97b4b11303cfb58c46894267c01b04d0ac4e54081b8f5a9564fa274a9cc07c"}
Sep 16 19:22:41.462617 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:41.462225    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" event={"ID":"a682bdb3-a58f-4d64-86e2-0f8cf1993a37","Type":"ContainerStarted","Data":"931b62c34c070a6126d6f7d3c364b972d6d17687286fbaa1736ddfbc951f7585"}
Sep 16 19:22:41.462617 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:41.462473    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:41.463690 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:41.463667    2584 generic.go:356] "Generic (PLEG): container finished" podID="43ad6e3e-f147-453d-9609-0f44347ce117" containerID="2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623" exitCode=0
Sep 16 19:22:41.463774 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:41.463739    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" event={"ID":"43ad6e3e-f147-453d-9609-0f44347ce117","Type":"ContainerDied","Data":"2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623"}
Sep 16 19:22:41.485069 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:41.484897    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podStartSLOduration=2.705021763 podStartE2EDuration="6.484879384s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.779857621s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:41.484800479 +0000 UTC m=+2292.151739625" watchObservedRunningTime="2026-09-16 19:22:41.484879384 +0000 UTC m=+2292.151818533"
Sep 16 19:22:42.470129 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:42.470095    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" event={"ID":"43ad6e3e-f147-453d-9609-0f44347ce117","Type":"ContainerStarted","Data":"b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4"}
Sep 16 19:22:42.470538 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:42.470153    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" event={"ID":"43ad6e3e-f147-453d-9609-0f44347ce117","Type":"ContainerStarted","Data":"ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab"}
Sep 16 19:22:42.470538 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:42.470417    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:42.470673 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:42.470651    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:42.470845 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:42.470826    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:42.471871 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:42.471847    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:22:42.472033 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:42.472012    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:22:42.490595 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:42.490517    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podStartSLOduration=2.544384877 podStartE2EDuration="6.490502918s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.946118041s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:42.490306442 +0000 UTC m=+2293.157245605" watchObservedRunningTime="2026-09-16 19:22:42.490502918 +0000 UTC m=+2293.157442264"
Sep 16 19:22:43.475095 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:43.475060    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:22:43.475527 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:43.475215    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:22:48.480917 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:48.480885    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:22:48.481432 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:48.481408    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:22:48.481867 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:48.481827    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:22:48.481998 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:48.481886    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:22:58.482108 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:58.482071    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:22:58.482609 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:22:58.482071    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:23:08.482239 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:08.482196    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:23:08.482645 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:08.482201    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:23:18.482677 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:18.482638    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:23:18.483089 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:18.482637    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:23:28.482413 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:28.482330    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:23:28.482793 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:28.482331    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:23:38.482081 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:38.482044    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:23:38.482541 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:38.482041    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:23:48.482673 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:48.482636    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:23:48.483085 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:48.482636    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:23:49.819858 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:49.819785    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:23:58.483036 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:58.482998    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:23:59.818331 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:23:59.818304    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:24:05.788610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:05.788575    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"]
Sep 16 19:24:05.789113 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:05.788819    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" containerID="cri-o://931b62c34c070a6126d6f7d3c364b972d6d17687286fbaa1736ddfbc951f7585" gracePeriod=30
Sep 16 19:24:05.789113 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:05.788919    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kube-rbac-proxy" containerID="cri-o://0e97b4b11303cfb58c46894267c01b04d0ac4e54081b8f5a9564fa274a9cc07c" gracePeriod=30
Sep 16 19:24:05.826508 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:05.826476    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"]
Sep 16 19:24:05.826736 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:05.826709    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" containerID="cri-o://ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab" gracePeriod=30
Sep 16 19:24:05.826797 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:05.826747    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kube-rbac-proxy" containerID="cri-o://b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4" gracePeriod=30
Sep 16 19:24:06.864563 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:06.864529    2584 generic.go:356] "Generic (PLEG): container finished" podID="43ad6e3e-f147-453d-9609-0f44347ce117" containerID="b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4" exitCode=2
Sep 16 19:24:06.864969 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:06.864610    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" event={"ID":"43ad6e3e-f147-453d-9609-0f44347ce117","Type":"ContainerDied","Data":"b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4"}
Sep 16 19:24:06.866615 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:06.866591    2584 generic.go:356] "Generic (PLEG): container finished" podID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerID="0e97b4b11303cfb58c46894267c01b04d0ac4e54081b8f5a9564fa274a9cc07c" exitCode=2
Sep 16 19:24:06.866729 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:06.866646    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" event={"ID":"a682bdb3-a58f-4d64-86e2-0f8cf1993a37","Type":"ContainerDied","Data":"0e97b4b11303cfb58c46894267c01b04d0ac4e54081b8f5a9564fa274a9cc07c"}
Sep 16 19:24:08.475514 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:08.475470    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 16 19:24:08.475910 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:08.475470    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 16 19:24:08.481851 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:08.481821    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 16 19:24:09.817343 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:09.817300    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 16 19:24:10.847887 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.847860    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:24:10.888903 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.888867    2584 generic.go:356] "Generic (PLEG): container finished" podID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerID="931b62c34c070a6126d6f7d3c364b972d6d17687286fbaa1736ddfbc951f7585" exitCode=0
Sep 16 19:24:10.889078 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.888947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" event={"ID":"a682bdb3-a58f-4d64-86e2-0f8cf1993a37","Type":"ContainerDied","Data":"931b62c34c070a6126d6f7d3c364b972d6d17687286fbaa1736ddfbc951f7585"}
Sep 16 19:24:10.890933 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.890911    2584 generic.go:356] "Generic (PLEG): container finished" podID="43ad6e3e-f147-453d-9609-0f44347ce117" containerID="ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab" exitCode=0
Sep 16 19:24:10.891048 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.890977    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"
Sep 16 19:24:10.891048 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.890994    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" event={"ID":"43ad6e3e-f147-453d-9609-0f44347ce117","Type":"ContainerDied","Data":"ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab"}
Sep 16 19:24:10.891048 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.891038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh" event={"ID":"43ad6e3e-f147-453d-9609-0f44347ce117","Type":"ContainerDied","Data":"36c871efafdffc7b6a134689969d9c0c972e3a8f782c5529c8c50e5266d0f298"}
Sep 16 19:24:10.891151 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.891070    2584 scope.go:117] "RemoveContainer" containerID="b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4"
Sep 16 19:24:10.900429 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.900388    2584 scope.go:117] "RemoveContainer" containerID="ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab"
Sep 16 19:24:10.908393 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.908373    2584 scope.go:117] "RemoveContainer" containerID="2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623"
Sep 16 19:24:10.917313 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.917288    2584 scope.go:117] "RemoveContainer" containerID="b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4"
Sep 16 19:24:10.917632 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:24:10.917611    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4\": container with ID starting with b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4 not found: ID does not exist" containerID="b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4"
Sep 16 19:24:10.917687 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.917644    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4"} err="failed to get container status \"b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4\": rpc error: code = NotFound desc = could not find container \"b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4\": container with ID starting with b8c0dbabf0d97f482d01dbbb32ba0b2f1184227e1867d93b57b837be642575d4 not found: ID does not exist"
Sep 16 19:24:10.917687 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.917662    2584 scope.go:117] "RemoveContainer" containerID="ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab"
Sep 16 19:24:10.917935 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:24:10.917907    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab\": container with ID starting with ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab not found: ID does not exist" containerID="ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab"
Sep 16 19:24:10.918011 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.917943    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab"} err="failed to get container status \"ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab\": rpc error: code = NotFound desc = could not find container \"ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab\": container with ID starting with ad27f5bb7a6c675c46efe3a67c10568133b0d9922ba566da47131a15bd6d83ab not found: ID does not exist"
Sep 16 19:24:10.918011 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.917965    2584 scope.go:117] "RemoveContainer" containerID="2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623"
Sep 16 19:24:10.918212 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:24:10.918196    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623\": container with ID starting with 2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623 not found: ID does not exist" containerID="2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623"
Sep 16 19:24:10.918255 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.918215    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623"} err="failed to get container status \"2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623\": rpc error: code = NotFound desc = could not find container \"2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623\": container with ID starting with 2daeec5d5e8827919fc1563fd79f9733e336067fa484e89c80977a37e91e0623 not found: ID does not exist"
Sep 16 19:24:10.950905 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:10.950876    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:24:11.021689 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.021649    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdlp9\" (UniqueName: \"kubernetes.io/projected/43ad6e3e-f147-453d-9609-0f44347ce117-kube-api-access-cdlp9\") pod \"43ad6e3e-f147-453d-9609-0f44347ce117\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") "
Sep 16 19:24:11.021900 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.021735    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-kube-rbac-proxy-sar-config\") pod \"43ad6e3e-f147-453d-9609-0f44347ce117\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") "
Sep 16 19:24:11.021900 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.021848    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-cabundle-cert\") pod \"43ad6e3e-f147-453d-9609-0f44347ce117\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") "
Sep 16 19:24:11.022032 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.021906    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ad6e3e-f147-453d-9609-0f44347ce117-kserve-provision-location\") pod \"43ad6e3e-f147-453d-9609-0f44347ce117\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") "
Sep 16 19:24:11.022032 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.021944    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ad6e3e-f147-453d-9609-0f44347ce117-proxy-tls\") pod \"43ad6e3e-f147-453d-9609-0f44347ce117\" (UID: \"43ad6e3e-f147-453d-9609-0f44347ce117\") "
Sep 16 19:24:11.022176 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.022058    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43ad6e3e-f147-453d-9609-0f44347ce117" (UID: "43ad6e3e-f147-453d-9609-0f44347ce117"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:11.022237 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.022217    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43ad6e3e-f147-453d-9609-0f44347ce117" (UID: "43ad6e3e-f147-453d-9609-0f44347ce117"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:11.022310 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.022288    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ad6e3e-f147-453d-9609-0f44347ce117-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43ad6e3e-f147-453d-9609-0f44347ce117" (UID: "43ad6e3e-f147-453d-9609-0f44347ce117"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:11.023893 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.023873    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ad6e3e-f147-453d-9609-0f44347ce117-kube-api-access-cdlp9" (OuterVolumeSpecName: "kube-api-access-cdlp9") pod "43ad6e3e-f147-453d-9609-0f44347ce117" (UID: "43ad6e3e-f147-453d-9609-0f44347ce117"). InnerVolumeSpecName "kube-api-access-cdlp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:11.023987 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.023968    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ad6e3e-f147-453d-9609-0f44347ce117-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43ad6e3e-f147-453d-9609-0f44347ce117" (UID: "43ad6e3e-f147-453d-9609-0f44347ce117"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:11.122835 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.122730    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-proxy-tls\") pod \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") "
Sep 16 19:24:11.122835 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.122777    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kserve-provision-location\") pod \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") "
Sep 16 19:24:11.122835 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.122811    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-cabundle-cert\") pod \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") "
Sep 16 19:24:11.123116 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.122888    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-rbac-proxy-sar-config\") pod \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") "
Sep 16 19:24:11.123116 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.122916    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gc4j\" (UniqueName: \"kubernetes.io/projected/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-api-access-8gc4j\") pod \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\" (UID: \"a682bdb3-a58f-4d64-86e2-0f8cf1993a37\") "
Sep 16 19:24:11.123116 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.123052    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a682bdb3-a58f-4d64-86e2-0f8cf1993a37" (UID: "a682bdb3-a58f-4d64-86e2-0f8cf1993a37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:11.123235 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.123184    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a682bdb3-a58f-4d64-86e2-0f8cf1993a37" (UID: "a682bdb3-a58f-4d64-86e2-0f8cf1993a37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:11.123308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.123234    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a682bdb3-a58f-4d64-86e2-0f8cf1993a37" (UID: "a682bdb3-a58f-4d64-86e2-0f8cf1993a37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:11.123308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.123241    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43ad6e3e-f147-453d-9609-0f44347ce117-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.123308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.123283    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.123308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.123299    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43ad6e3e-f147-453d-9609-0f44347ce117-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.123308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.123309    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdlp9\" (UniqueName: \"kubernetes.io/projected/43ad6e3e-f147-453d-9609-0f44347ce117-kube-api-access-cdlp9\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.123479 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.123320    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.123479 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.123332    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43ad6e3e-f147-453d-9609-0f44347ce117-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.124860 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.124842    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a682bdb3-a58f-4d64-86e2-0f8cf1993a37" (UID: "a682bdb3-a58f-4d64-86e2-0f8cf1993a37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:11.125000 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.124977    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-api-access-8gc4j" (OuterVolumeSpecName: "kube-api-access-8gc4j") pod "a682bdb3-a58f-4d64-86e2-0f8cf1993a37" (UID: "a682bdb3-a58f-4d64-86e2-0f8cf1993a37"). InnerVolumeSpecName "kube-api-access-8gc4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:11.215608 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.215573    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"]
Sep 16 19:24:11.220022 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.219992    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-97qfh"]
Sep 16 19:24:11.223839 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.223820    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.223912 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.223844    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.223912 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.223854    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.223912 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.223863    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gc4j\" (UniqueName: \"kubernetes.io/projected/a682bdb3-a58f-4d64-86e2-0f8cf1993a37-kube-api-access-8gc4j\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:24:11.815940 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.815907    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" path="/var/lib/kubelet/pods/43ad6e3e-f147-453d-9609-0f44347ce117/volumes"
Sep 16 19:24:11.897413 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.897325    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"
Sep 16 19:24:11.897413 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.897336    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7" event={"ID":"a682bdb3-a58f-4d64-86e2-0f8cf1993a37","Type":"ContainerDied","Data":"8ba9aeca8c4e92f671ec8bfa1c60eb17937b21bd0e6f67a440880bea36e4f58d"}
Sep 16 19:24:11.897413 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.897411    2584 scope.go:117] "RemoveContainer" containerID="0e97b4b11303cfb58c46894267c01b04d0ac4e54081b8f5a9564fa274a9cc07c"
Sep 16 19:24:11.908335 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.908307    2584 scope.go:117] "RemoveContainer" containerID="931b62c34c070a6126d6f7d3c364b972d6d17687286fbaa1736ddfbc951f7585"
Sep 16 19:24:11.916499 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.916478    2584 scope.go:117] "RemoveContainer" containerID="a467adaa49560db98149eff1aecc16e511403495c5d4799e9f436d0a648075da"
Sep 16 19:24:11.924648 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.924627    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"]
Sep 16 19:24:11.928533 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:11.928507    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-84ff76f474-zv7d7"]
Sep 16 19:24:13.817438 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:13.817399    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" path="/var/lib/kubelet/pods/a682bdb3-a58f-4d64-86e2-0f8cf1993a37/volumes"
Sep 16 19:24:16.046294 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046248    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"]
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046582    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046594    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046610    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kube-rbac-proxy"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046615    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kube-rbac-proxy"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046626    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046631    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046643    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="storage-initializer"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046648    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="storage-initializer"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046655    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kube-rbac-proxy"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046660    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kube-rbac-proxy"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046668    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="storage-initializer"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046674    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="storage-initializer"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046723    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kube-rbac-proxy"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046732    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kube-rbac-proxy"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046738    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ad6e3e-f147-453d-9609-0f44347ce117" containerName="kserve-container"
Sep 16 19:24:16.046741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.046742    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a682bdb3-a58f-4d64-86e2-0f8cf1993a37" containerName="kserve-container"
Sep 16 19:24:16.051687 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.051659    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.054543 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.054521    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 16 19:24:16.054543 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.054535    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:24:16.054737 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.054535    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:24:16.054737 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.054631    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 19:24:16.055613 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.055590    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:24:16.055723 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.055618    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:24:16.055723 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.055633    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:24:16.066056 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.066035    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"]
Sep 16 19:24:16.163250 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.163212    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.163413 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.163257    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598af449-517a-4dd9-8ac4-06bb472a8e1a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.163413 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.163388    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.163501 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.163430    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm6dt\" (UniqueName: \"kubernetes.io/projected/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-api-access-fm6dt\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.163501 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.163460    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598af449-517a-4dd9-8ac4-06bb472a8e1a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.263999 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.263967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.264153 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.264011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm6dt\" (UniqueName: \"kubernetes.io/projected/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-api-access-fm6dt\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.264153 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.264060    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598af449-517a-4dd9-8ac4-06bb472a8e1a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.264153 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.264104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.264153 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.264137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598af449-517a-4dd9-8ac4-06bb472a8e1a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.264698 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.264665    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598af449-517a-4dd9-8ac4-06bb472a8e1a-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.264865 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.264846    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.264945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.264890    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.266638 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.266620    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598af449-517a-4dd9-8ac4-06bb472a8e1a-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.272927 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.272897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm6dt\" (UniqueName: \"kubernetes.io/projected/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-api-access-fm6dt\") pod \"isvc-lightgbm-predictor-54f85ddf88-vq2ds\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.362335 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.362248    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:16.636041 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.636015    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"]
Sep 16 19:24:16.638356 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:24:16.638322    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598af449_517a_4dd9_8ac4_06bb472a8e1a.slice/crio-0b013821e37d4f746d45ca2963475d1d785832af7e7be7eb6ef8000671186dd3 WatchSource:0}: Error finding container 0b013821e37d4f746d45ca2963475d1d785832af7e7be7eb6ef8000671186dd3: Status 404 returned error can't find the container with id 0b013821e37d4f746d45ca2963475d1d785832af7e7be7eb6ef8000671186dd3
Sep 16 19:24:16.925038 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.925005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" event={"ID":"598af449-517a-4dd9-8ac4-06bb472a8e1a","Type":"ContainerStarted","Data":"c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61"}
Sep 16 19:24:16.925200 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:16.925065    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" event={"ID":"598af449-517a-4dd9-8ac4-06bb472a8e1a","Type":"ContainerStarted","Data":"0b013821e37d4f746d45ca2963475d1d785832af7e7be7eb6ef8000671186dd3"}
Sep 16 19:24:20.944511 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:20.944480    2584 generic.go:356] "Generic (PLEG): container finished" podID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerID="c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61" exitCode=0
Sep 16 19:24:20.944858 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:20.944525    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" event={"ID":"598af449-517a-4dd9-8ac4-06bb472a8e1a","Type":"ContainerDied","Data":"c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61"}
Sep 16 19:24:20.945844 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:20.945827    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:24:27.986383 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:27.986347    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" event={"ID":"598af449-517a-4dd9-8ac4-06bb472a8e1a","Type":"ContainerStarted","Data":"42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283"}
Sep 16 19:24:27.986753 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:27.986410    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" event={"ID":"598af449-517a-4dd9-8ac4-06bb472a8e1a","Type":"ContainerStarted","Data":"7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134"}
Sep 16 19:24:27.986753 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:27.986596    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:28.008914 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:28.008860    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podStartSLOduration=1.767237873 podStartE2EDuration="12.00884128s" totalImagesPullingTime="6.121439694s" totalInitContainerRuntime="4.120163713s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:28.008215192 +0000 UTC m=+2398.675154355" watchObservedRunningTime="2026-09-16 19:24:28.00884128 +0000 UTC m=+2398.675780429"
Sep 16 19:24:28.991332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:28.991292    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:28.992489 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:28.992462    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 16 19:24:29.995161 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:29.995127    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 16 19:24:34.999262 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:34.999231    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:24:34.999742 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:34.999717    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 16 19:24:45.000073 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:45.000033    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 16 19:24:49.742152 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.742115    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"]
Sep 16 19:24:49.746374 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.746353    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.750057 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.750030    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 16 19:24:49.750184 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.750071    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:24:49.761174 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.761154    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"]
Sep 16 19:24:49.813130 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.813099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.813292 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.813165    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.813292 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.813221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dc8e25a-429a-445b-99d0-eba35e2a69d5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.813422 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.813316    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgc8\" (UniqueName: \"kubernetes.io/projected/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-api-access-6zgc8\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.813422 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.813339    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.914734 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.914708    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.914905 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.914748    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.914905 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.914776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dc8e25a-429a-445b-99d0-eba35e2a69d5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.914905 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.914837    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgc8\" (UniqueName: \"kubernetes.io/projected/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-api-access-6zgc8\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.914905 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.914852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.915209 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.915186    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.915508 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.915483    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.915508 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.915497    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.917359 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.917333    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dc8e25a-429a-445b-99d0-eba35e2a69d5-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:49.933586 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:49.933566    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgc8\" (UniqueName: \"kubernetes.io/projected/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-api-access-6zgc8\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-vrcs9\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:50.056562 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:50.056482    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:24:50.338194 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:50.338165    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"]
Sep 16 19:24:50.339739 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:24:50.339711    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc8e25a_429a_445b_99d0_eba35e2a69d5.slice/crio-8849ee9dc73b143d163654528215ce601c82779fb037dd6ba3a7c816423d5967 WatchSource:0}: Error finding container 8849ee9dc73b143d163654528215ce601c82779fb037dd6ba3a7c816423d5967: Status 404 returned error can't find the container with id 8849ee9dc73b143d163654528215ce601c82779fb037dd6ba3a7c816423d5967
Sep 16 19:24:51.091536 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:51.091500    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" event={"ID":"3dc8e25a-429a-445b-99d0-eba35e2a69d5","Type":"ContainerStarted","Data":"0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014"}
Sep 16 19:24:51.091897 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:51.091564    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" event={"ID":"3dc8e25a-429a-445b-99d0-eba35e2a69d5","Type":"ContainerStarted","Data":"8849ee9dc73b143d163654528215ce601c82779fb037dd6ba3a7c816423d5967"}
Sep 16 19:24:55.000111 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:55.000076    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 16 19:24:55.111959 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:55.111877    2584 generic.go:356] "Generic (PLEG): container finished" podID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerID="0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014" exitCode=0
Sep 16 19:24:55.112097 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:24:55.111954    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" event={"ID":"3dc8e25a-429a-445b-99d0-eba35e2a69d5","Type":"ContainerDied","Data":"0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014"}
Sep 16 19:25:04.999753 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:04.999713    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 16 19:25:13.209280 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:13.209236    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" event={"ID":"3dc8e25a-429a-445b-99d0-eba35e2a69d5","Type":"ContainerStarted","Data":"1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9"}
Sep 16 19:25:13.209689 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:13.209318    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" event={"ID":"3dc8e25a-429a-445b-99d0-eba35e2a69d5","Type":"ContainerStarted","Data":"a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20"}
Sep 16 19:25:13.209689 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:13.209505    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:25:13.234647 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:13.234596    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podStartSLOduration=3.091746984 podStartE2EDuration="24.234580719s" totalImagesPullingTime="17.054143149s" totalInitContainerRuntime="4.088690586s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:13.233196468 +0000 UTC m=+2443.900135632" watchObservedRunningTime="2026-09-16 19:25:13.234580719 +0000 UTC m=+2443.901519864"
Sep 16 19:25:14.215469 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:14.215440    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:25:14.216752 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:14.216726    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 16 19:25:14.999777 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:14.999732    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 16 19:25:15.219681 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:15.219634    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 16 19:25:20.224010 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:20.223980    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:25:20.224546 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:20.224517    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 16 19:25:24.999878 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:24.999842    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 16 19:25:30.225283 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:30.225225    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 16 19:25:35.000024 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:34.999981    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 16 19:25:40.224577 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:40.224540    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 16 19:25:45.000835 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:45.000806    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:25:46.333585 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:46.333548    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"]
Sep 16 19:25:46.333952 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:46.333822    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container" containerID="cri-o://7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134" gracePeriod=30
Sep 16 19:25:46.333952 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:46.333905    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kube-rbac-proxy" containerID="cri-o://42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283" gracePeriod=30
Sep 16 19:25:47.366430 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:47.366395    2584 generic.go:356] "Generic (PLEG): container finished" podID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerID="42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283" exitCode=2
Sep 16 19:25:47.366798 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:47.366469    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" event={"ID":"598af449-517a-4dd9-8ac4-06bb472a8e1a","Type":"ContainerDied","Data":"42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283"}
Sep 16 19:25:49.995966 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:49.995923    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Sep 16 19:25:50.225133 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.225088    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 16 19:25:50.572438 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.572415    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:25:50.647929 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.647895    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598af449-517a-4dd9-8ac4-06bb472a8e1a-kserve-provision-location\") pod \"598af449-517a-4dd9-8ac4-06bb472a8e1a\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") "
Sep 16 19:25:50.647929 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.647937    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-cabundle-cert\") pod \"598af449-517a-4dd9-8ac4-06bb472a8e1a\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") "
Sep 16 19:25:50.648147 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.647961    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-rbac-proxy-sar-config\") pod \"598af449-517a-4dd9-8ac4-06bb472a8e1a\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") "
Sep 16 19:25:50.648147 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.647981    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598af449-517a-4dd9-8ac4-06bb472a8e1a-proxy-tls\") pod \"598af449-517a-4dd9-8ac4-06bb472a8e1a\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") "
Sep 16 19:25:50.648147 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.648048    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm6dt\" (UniqueName: \"kubernetes.io/projected/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-api-access-fm6dt\") pod \"598af449-517a-4dd9-8ac4-06bb472a8e1a\" (UID: \"598af449-517a-4dd9-8ac4-06bb472a8e1a\") "
Sep 16 19:25:50.648371 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.648348    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "598af449-517a-4dd9-8ac4-06bb472a8e1a" (UID: "598af449-517a-4dd9-8ac4-06bb472a8e1a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:50.648420 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.648345    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598af449-517a-4dd9-8ac4-06bb472a8e1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "598af449-517a-4dd9-8ac4-06bb472a8e1a" (UID: "598af449-517a-4dd9-8ac4-06bb472a8e1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:50.648420 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.648355    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "598af449-517a-4dd9-8ac4-06bb472a8e1a" (UID: "598af449-517a-4dd9-8ac4-06bb472a8e1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:50.650175 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.650143    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-api-access-fm6dt" (OuterVolumeSpecName: "kube-api-access-fm6dt") pod "598af449-517a-4dd9-8ac4-06bb472a8e1a" (UID: "598af449-517a-4dd9-8ac4-06bb472a8e1a"). InnerVolumeSpecName "kube-api-access-fm6dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:50.650175 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.650155    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598af449-517a-4dd9-8ac4-06bb472a8e1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "598af449-517a-4dd9-8ac4-06bb472a8e1a" (UID: "598af449-517a-4dd9-8ac4-06bb472a8e1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:50.749086 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.749049    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm6dt\" (UniqueName: \"kubernetes.io/projected/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-api-access-fm6dt\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:25:50.749086 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.749084    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/598af449-517a-4dd9-8ac4-06bb472a8e1a-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:25:50.749086 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.749095    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:25:50.749358 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.749106    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/598af449-517a-4dd9-8ac4-06bb472a8e1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:25:50.749358 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:50.749115    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/598af449-517a-4dd9-8ac4-06bb472a8e1a-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:25:51.386503 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.386468    2584 generic.go:356] "Generic (PLEG): container finished" podID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerID="7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134" exitCode=0
Sep 16 19:25:51.387021 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.386521    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"
Sep 16 19:25:51.387021 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.386555    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" event={"ID":"598af449-517a-4dd9-8ac4-06bb472a8e1a","Type":"ContainerDied","Data":"7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134"}
Sep 16 19:25:51.387021 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.386619    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds" event={"ID":"598af449-517a-4dd9-8ac4-06bb472a8e1a","Type":"ContainerDied","Data":"0b013821e37d4f746d45ca2963475d1d785832af7e7be7eb6ef8000671186dd3"}
Sep 16 19:25:51.387021 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.386647    2584 scope.go:117] "RemoveContainer" containerID="42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283"
Sep 16 19:25:51.395673 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.395658    2584 scope.go:117] "RemoveContainer" containerID="7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134"
Sep 16 19:25:51.403477 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.403462    2584 scope.go:117] "RemoveContainer" containerID="c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61"
Sep 16 19:25:51.412439 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.412409    2584 scope.go:117] "RemoveContainer" containerID="42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283"
Sep 16 19:25:51.412807 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:25:51.412785    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283\": container with ID starting with 42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283 not found: ID does not exist" containerID="42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283"
Sep 16 19:25:51.412916 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.412817    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283"} err="failed to get container status \"42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283\": rpc error: code = NotFound desc = could not find container \"42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283\": container with ID starting with 42e6cede0bb17bdf4be4736a7a1bba0e38342ab80d5fc9b3c2af33132d3af283 not found: ID does not exist"
Sep 16 19:25:51.412916 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.412847    2584 scope.go:117] "RemoveContainer" containerID="7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134"
Sep 16 19:25:51.413647 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:25:51.413495    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134\": container with ID starting with 7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134 not found: ID does not exist" containerID="7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134"
Sep 16 19:25:51.413647 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.413526    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134"} err="failed to get container status \"7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134\": rpc error: code = NotFound desc = could not find container \"7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134\": container with ID starting with 7a47cff0ce13e986c3b0051dd09ad3e77f86fd0ca71f71de32543d21b5baf134 not found: ID does not exist"
Sep 16 19:25:51.413647 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.413546    2584 scope.go:117] "RemoveContainer" containerID="c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61"
Sep 16 19:25:51.414144 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:25:51.414090    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61\": container with ID starting with c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61 not found: ID does not exist" containerID="c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61"
Sep 16 19:25:51.414233 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.414168    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61"} err="failed to get container status \"c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61\": rpc error: code = NotFound desc = could not find container \"c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61\": container with ID starting with c47d78e5ea0300bd1a02e8fba4cf87af334b179e4150a2ad34893518ad928b61 not found: ID does not exist"
Sep 16 19:25:51.414885 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.414860    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"]
Sep 16 19:25:51.417227 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.417200    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vq2ds"]
Sep 16 19:25:51.815813 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:25:51.815780    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" path="/var/lib/kubelet/pods/598af449-517a-4dd9-8ac4-06bb472a8e1a/volumes"
Sep 16 19:26:00.224614 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:00.224577    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 16 19:26:10.224930 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:10.224890    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 16 19:26:20.225705 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:20.225674    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:26:29.789286 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:29.789192    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"]
Sep 16 19:26:29.789796 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:29.789454    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" containerID="cri-o://a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20" gracePeriod=30
Sep 16 19:26:29.789796 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:29.789521    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kube-rbac-proxy" containerID="cri-o://1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9" gracePeriod=30
Sep 16 19:26:30.220444 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:30.220405    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 16 19:26:30.224996 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:30.224969    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 16 19:26:30.570093 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:30.570016    2584 generic.go:356] "Generic (PLEG): container finished" podID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerID="1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9" exitCode=2
Sep 16 19:26:30.570093 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:30.570063    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" event={"ID":"3dc8e25a-429a-445b-99d0-eba35e2a69d5","Type":"ContainerDied","Data":"1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9"}
Sep 16 19:26:33.141218 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.141190    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:26:33.178493 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.178415    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-rbac-proxy-sar-config\") pod \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") "
Sep 16 19:26:33.178493 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.178453    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zgc8\" (UniqueName: \"kubernetes.io/projected/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-api-access-6zgc8\") pod \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") "
Sep 16 19:26:33.178493 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.178482    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-cabundle-cert\") pod \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") "
Sep 16 19:26:33.178766 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.178503    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dc8e25a-429a-445b-99d0-eba35e2a69d5-proxy-tls\") pod \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") "
Sep 16 19:26:33.178766 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.178532    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kserve-provision-location\") pod \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\" (UID: \"3dc8e25a-429a-445b-99d0-eba35e2a69d5\") "
Sep 16 19:26:33.178911 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.178871    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3dc8e25a-429a-445b-99d0-eba35e2a69d5" (UID: "3dc8e25a-429a-445b-99d0-eba35e2a69d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:33.178911 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.178871    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3dc8e25a-429a-445b-99d0-eba35e2a69d5" (UID: "3dc8e25a-429a-445b-99d0-eba35e2a69d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:33.178911 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.178880    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3dc8e25a-429a-445b-99d0-eba35e2a69d5" (UID: "3dc8e25a-429a-445b-99d0-eba35e2a69d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:33.180639 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.180612    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc8e25a-429a-445b-99d0-eba35e2a69d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3dc8e25a-429a-445b-99d0-eba35e2a69d5" (UID: "3dc8e25a-429a-445b-99d0-eba35e2a69d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:33.180763 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.180737    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-api-access-6zgc8" (OuterVolumeSpecName: "kube-api-access-6zgc8") pod "3dc8e25a-429a-445b-99d0-eba35e2a69d5" (UID: "3dc8e25a-429a-445b-99d0-eba35e2a69d5"). InnerVolumeSpecName "kube-api-access-6zgc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:33.279238 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.279199    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:26:33.279238 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.279232    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgc8\" (UniqueName: \"kubernetes.io/projected/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kube-api-access-6zgc8\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:26:33.279238 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.279242    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc8e25a-429a-445b-99d0-eba35e2a69d5-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:26:33.279511 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.279252    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dc8e25a-429a-445b-99d0-eba35e2a69d5-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:26:33.279511 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.279260    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc8e25a-429a-445b-99d0-eba35e2a69d5-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:26:33.584976 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.584880    2584 generic.go:356] "Generic (PLEG): container finished" podID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerID="a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20" exitCode=0
Sep 16 19:26:33.584976 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.584944    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"
Sep 16 19:26:33.585157 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.584973    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" event={"ID":"3dc8e25a-429a-445b-99d0-eba35e2a69d5","Type":"ContainerDied","Data":"a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20"}
Sep 16 19:26:33.585157 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.585037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9" event={"ID":"3dc8e25a-429a-445b-99d0-eba35e2a69d5","Type":"ContainerDied","Data":"8849ee9dc73b143d163654528215ce601c82779fb037dd6ba3a7c816423d5967"}
Sep 16 19:26:33.585157 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.585064    2584 scope.go:117] "RemoveContainer" containerID="1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9"
Sep 16 19:26:33.594473 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.594454    2584 scope.go:117] "RemoveContainer" containerID="a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20"
Sep 16 19:26:33.602216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.602192    2584 scope.go:117] "RemoveContainer" containerID="0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014"
Sep 16 19:26:33.607743 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.607720    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"]
Sep 16 19:26:33.610426 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.610406    2584 scope.go:117] "RemoveContainer" containerID="1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9"
Sep 16 19:26:33.611069 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:26:33.611037    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9\": container with ID starting with 1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9 not found: ID does not exist" containerID="1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9"
Sep 16 19:26:33.611243 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.611080    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9"} err="failed to get container status \"1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9\": rpc error: code = NotFound desc = could not find container \"1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9\": container with ID starting with 1408bc47112dff5c870165fd6a6a4e23d0bd9f25f9e2619eb18bc082001bddb9 not found: ID does not exist"
Sep 16 19:26:33.611243 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.611107    2584 scope.go:117] "RemoveContainer" containerID="a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20"
Sep 16 19:26:33.614005 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.613975    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-vrcs9"]
Sep 16 19:26:33.617287 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:26:33.617239    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20\": container with ID starting with a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20 not found: ID does not exist" containerID="a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20"
Sep 16 19:26:33.617397 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.617333    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20"} err="failed to get container status \"a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20\": rpc error: code = NotFound desc = could not find container \"a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20\": container with ID starting with a32f258afac055456b07927b70e0e00dc16141ab7f3dbf91bb74c5945d799f20 not found: ID does not exist"
Sep 16 19:26:33.617397 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.617358    2584 scope.go:117] "RemoveContainer" containerID="0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014"
Sep 16 19:26:33.617851 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:26:33.617832    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014\": container with ID starting with 0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014 not found: ID does not exist" containerID="0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014"
Sep 16 19:26:33.617924 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.617856    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014"} err="failed to get container status \"0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014\": rpc error: code = NotFound desc = could not find container \"0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014\": container with ID starting with 0a587c6e22621658b8845bf90bab2171755111215d7cdacc5f1b989d8837f014 not found: ID does not exist"
Sep 16 19:26:33.816674 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:26:33.816637    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" path="/var/lib/kubelet/pods/3dc8e25a-429a-445b-99d0-eba35e2a69d5/volumes"
Sep 16 19:27:40.238461 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238425    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"]
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238764    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="storage-initializer"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238776    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="storage-initializer"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238787    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238792    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238799    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kube-rbac-proxy"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238804    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kube-rbac-proxy"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238823    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kube-rbac-proxy"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238831    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kube-rbac-proxy"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238844    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="storage-initializer"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238849    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="storage-initializer"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238856    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238861    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238918    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kube-rbac-proxy"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238927    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kserve-container"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238935    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="598af449-517a-4dd9-8ac4-06bb472a8e1a" containerName="kserve-container"
Sep 16 19:27:40.238932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.238941    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dc8e25a-429a-445b-99d0-eba35e2a69d5" containerName="kube-rbac-proxy"
Sep 16 19:27:40.242038 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.242018    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.246575 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.246542    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:27:40.246699 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.246665    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:27:40.246699 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.246680    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:27:40.247535 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.247399    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 16 19:27:40.247535 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.247416    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:27:40.247535 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.247407    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:27:40.247535 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.247454    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:27:40.263378 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.263357    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"]
Sep 16 19:27:40.318289 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.318238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.318458 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.318319    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s2kd\" (UniqueName: \"kubernetes.io/projected/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-api-access-4s2kd\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.318458 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.318344    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.318458 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.318423    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5325af82-01a1-423c-aaa2-396a35e1ccc1-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.318458 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.318447    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5325af82-01a1-423c-aaa2-396a35e1ccc1-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.419336 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.419300    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2kd\" (UniqueName: \"kubernetes.io/projected/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-api-access-4s2kd\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.419336 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.419335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.419554 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.419473    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5325af82-01a1-423c-aaa2-396a35e1ccc1-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.419554 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.419519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5325af82-01a1-423c-aaa2-396a35e1ccc1-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.419658 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.419618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.419728 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:27:40.419700    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:27:40.419809 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:27:40.419783    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5325af82-01a1-423c-aaa2-396a35e1ccc1-proxy-tls podName:5325af82-01a1-423c-aaa2-396a35e1ccc1 nodeName:}" failed. No retries permitted until 2026-09-16 19:27:40.919761599 +0000 UTC m=+2591.586700728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5325af82-01a1-423c-aaa2-396a35e1ccc1-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-6gwqj" (UID: "5325af82-01a1-423c-aaa2-396a35e1ccc1") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 16 19:27:40.419940 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.419848    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5325af82-01a1-423c-aaa2-396a35e1ccc1-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.420100 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.420076    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.420234 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.420217    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.430821 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.430800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s2kd\" (UniqueName: \"kubernetes.io/projected/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-api-access-4s2kd\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.924603 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.924570    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5325af82-01a1-423c-aaa2-396a35e1ccc1-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:40.926984 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:40.926957    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5325af82-01a1-423c-aaa2-396a35e1ccc1-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6gwqj\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:41.152198 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:41.152165    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:41.426352 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:41.426325    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"]
Sep 16 19:27:41.428439 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:27:41.428414    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5325af82_01a1_423c_aaa2_396a35e1ccc1.slice/crio-b4f5d4ab3580257312b1dce4e35c6e20e33442d1058d6820632f46bde864c15b WatchSource:0}: Error finding container b4f5d4ab3580257312b1dce4e35c6e20e33442d1058d6820632f46bde864c15b: Status 404 returned error can't find the container with id b4f5d4ab3580257312b1dce4e35c6e20e33442d1058d6820632f46bde864c15b
Sep 16 19:27:41.885093 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:41.885016    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" event={"ID":"5325af82-01a1-423c-aaa2-396a35e1ccc1","Type":"ContainerStarted","Data":"9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75"}
Sep 16 19:27:41.885093 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:41.885071    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" event={"ID":"5325af82-01a1-423c-aaa2-396a35e1ccc1","Type":"ContainerStarted","Data":"b4f5d4ab3580257312b1dce4e35c6e20e33442d1058d6820632f46bde864c15b"}
Sep 16 19:27:45.904813 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:45.904733    2584 generic.go:356] "Generic (PLEG): container finished" podID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerID="9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75" exitCode=0
Sep 16 19:27:45.904813 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:45.904804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" event={"ID":"5325af82-01a1-423c-aaa2-396a35e1ccc1","Type":"ContainerDied","Data":"9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75"}
Sep 16 19:27:46.912135 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:46.912097    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" event={"ID":"5325af82-01a1-423c-aaa2-396a35e1ccc1","Type":"ContainerStarted","Data":"d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c"}
Sep 16 19:27:46.912635 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:46.912167    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" event={"ID":"5325af82-01a1-423c-aaa2-396a35e1ccc1","Type":"ContainerStarted","Data":"f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c"}
Sep 16 19:27:46.912635 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:46.912400    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:46.934481 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:46.934426    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podStartSLOduration=2.875728969 podStartE2EDuration="6.93440846s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.058679491s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:46.932490552 +0000 UTC m=+2597.599429716" watchObservedRunningTime="2026-09-16 19:27:46.93440846 +0000 UTC m=+2597.601347606"
Sep 16 19:27:47.916878 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:47.916846    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:47.918277 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:47.918242    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:27:48.921028 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:48.920967    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:27:53.926508 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:53.926429    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:27:53.926937 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:27:53.926912    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:28:03.927075 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:28:03.927036    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:28:13.927781 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:28:13.927741    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:28:23.927617 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:28:23.927574    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:28:33.927109 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:28:33.927066    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:28:43.927120 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:28:43.927081    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:28:53.927967 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:28:53.927932    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:29:00.328503 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:00.328472    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"]
Sep 16 19:29:00.328881 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:00.328693    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" containerID="cri-o://f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c" gracePeriod=30
Sep 16 19:29:00.328881 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:00.328738    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kube-rbac-proxy" containerID="cri-o://d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c" gracePeriod=30
Sep 16 19:29:01.253218 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:01.253184    2584 generic.go:356] "Generic (PLEG): container finished" podID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerID="d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c" exitCode=2
Sep 16 19:29:01.253422 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:01.253258    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" event={"ID":"5325af82-01a1-423c-aaa2-396a35e1ccc1","Type":"ContainerDied","Data":"d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c"}
Sep 16 19:29:03.921356 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:03.921324    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 16 19:29:03.927781 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:03.927751    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 16 19:29:04.070901 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.070874    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:29:04.118870 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.118836    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-cabundle-cert\") pod \"5325af82-01a1-423c-aaa2-396a35e1ccc1\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") "
Sep 16 19:29:04.119072 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.118896    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5325af82-01a1-423c-aaa2-396a35e1ccc1-kserve-provision-location\") pod \"5325af82-01a1-423c-aaa2-396a35e1ccc1\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") "
Sep 16 19:29:04.119072 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.118948    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s2kd\" (UniqueName: \"kubernetes.io/projected/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-api-access-4s2kd\") pod \"5325af82-01a1-423c-aaa2-396a35e1ccc1\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") "
Sep 16 19:29:04.119072 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.118986    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-rbac-proxy-sar-config\") pod \"5325af82-01a1-423c-aaa2-396a35e1ccc1\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") "
Sep 16 19:29:04.119072 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.119018    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5325af82-01a1-423c-aaa2-396a35e1ccc1-proxy-tls\") pod \"5325af82-01a1-423c-aaa2-396a35e1ccc1\" (UID: \"5325af82-01a1-423c-aaa2-396a35e1ccc1\") "
Sep 16 19:29:04.119310 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.119248    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5325af82-01a1-423c-aaa2-396a35e1ccc1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5325af82-01a1-423c-aaa2-396a35e1ccc1" (UID: "5325af82-01a1-423c-aaa2-396a35e1ccc1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:04.119310 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.119306    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5325af82-01a1-423c-aaa2-396a35e1ccc1" (UID: "5325af82-01a1-423c-aaa2-396a35e1ccc1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:04.119478 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.119367    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5325af82-01a1-423c-aaa2-396a35e1ccc1" (UID: "5325af82-01a1-423c-aaa2-396a35e1ccc1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:04.121340 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.121309    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5325af82-01a1-423c-aaa2-396a35e1ccc1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5325af82-01a1-423c-aaa2-396a35e1ccc1" (UID: "5325af82-01a1-423c-aaa2-396a35e1ccc1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:04.121468 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.121448    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-api-access-4s2kd" (OuterVolumeSpecName: "kube-api-access-4s2kd") pod "5325af82-01a1-423c-aaa2-396a35e1ccc1" (UID: "5325af82-01a1-423c-aaa2-396a35e1ccc1"). InnerVolumeSpecName "kube-api-access-4s2kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:04.220606 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.220549    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:29:04.220606 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.220598    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5325af82-01a1-423c-aaa2-396a35e1ccc1-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:29:04.220606 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.220611    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s2kd\" (UniqueName: \"kubernetes.io/projected/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-api-access-4s2kd\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:29:04.220606 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.220620    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5325af82-01a1-423c-aaa2-396a35e1ccc1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:29:04.220606 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.220630    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5325af82-01a1-423c-aaa2-396a35e1ccc1-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:29:04.267473 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.267433    2584 generic.go:356] "Generic (PLEG): container finished" podID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerID="f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c" exitCode=0
Sep 16 19:29:04.267683 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.267518    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"
Sep 16 19:29:04.267683 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.267518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" event={"ID":"5325af82-01a1-423c-aaa2-396a35e1ccc1","Type":"ContainerDied","Data":"f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c"}
Sep 16 19:29:04.267683 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.267604    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj" event={"ID":"5325af82-01a1-423c-aaa2-396a35e1ccc1","Type":"ContainerDied","Data":"b4f5d4ab3580257312b1dce4e35c6e20e33442d1058d6820632f46bde864c15b"}
Sep 16 19:29:04.267683 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.267632    2584 scope.go:117] "RemoveContainer" containerID="d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c"
Sep 16 19:29:04.276738 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.276716    2584 scope.go:117] "RemoveContainer" containerID="f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c"
Sep 16 19:29:04.285229 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.285206    2584 scope.go:117] "RemoveContainer" containerID="9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75"
Sep 16 19:29:04.293379 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.293351    2584 scope.go:117] "RemoveContainer" containerID="d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c"
Sep 16 19:29:04.293723 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:29:04.293702    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c\": container with ID starting with d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c not found: ID does not exist" containerID="d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c"
Sep 16 19:29:04.293807 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.293732    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c"} err="failed to get container status \"d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c\": rpc error: code = NotFound desc = could not find container \"d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c\": container with ID starting with d8921c44407a7b3e7c9e5d7f86d7eaa7369b413632e4c7eb0bb66d11fd7ebf4c not found: ID does not exist"
Sep 16 19:29:04.293807 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.293753    2584 scope.go:117] "RemoveContainer" containerID="f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c"
Sep 16 19:29:04.294038 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:29:04.294020    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c\": container with ID starting with f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c not found: ID does not exist" containerID="f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c"
Sep 16 19:29:04.294106 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.294048    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c"} err="failed to get container status \"f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c\": rpc error: code = NotFound desc = could not find container \"f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c\": container with ID starting with f8b24ab5f7aec6d68b344755e553ba727fa0027092b4451f3be1a82ca8460c1c not found: ID does not exist"
Sep 16 19:29:04.294106 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.294071    2584 scope.go:117] "RemoveContainer" containerID="9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75"
Sep 16 19:29:04.294376 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:29:04.294352    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75\": container with ID starting with 9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75 not found: ID does not exist" containerID="9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75"
Sep 16 19:29:04.294520 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.294379    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75"} err="failed to get container status \"9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75\": rpc error: code = NotFound desc = could not find container \"9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75\": container with ID starting with 9f3acc64ab874a78926a11149fd4ca5c7c82e86bf17fb73175786d0e0d688e75 not found: ID does not exist"
Sep 16 19:29:04.297850 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.297824    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"]
Sep 16 19:29:04.302619 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:04.302592    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6gwqj"]
Sep 16 19:29:05.556394 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.556353    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"]
Sep 16 19:29:05.556762 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.556727    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container"
Sep 16 19:29:05.556762 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.556739    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container"
Sep 16 19:29:05.556762 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.556753    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kube-rbac-proxy"
Sep 16 19:29:05.556762 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.556761    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kube-rbac-proxy"
Sep 16 19:29:05.556907 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.556772    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="storage-initializer"
Sep 16 19:29:05.556907 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.556778    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="storage-initializer"
Sep 16 19:29:05.556907 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.556867    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kserve-container"
Sep 16 19:29:05.556907 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.556877    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" containerName="kube-rbac-proxy"
Sep 16 19:29:05.561673 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.561644    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.566380 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.566354    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:29:05.566534 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.566440    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 16 19:29:05.566534 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.566462    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:29:05.567420 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.567402    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:29:05.567558 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.567441    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 16 19:29:05.567620 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.567560    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:29:05.575205 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.575180    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"]
Sep 16 19:29:05.633222 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.633187    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/750732c0-2b7c-452a-9155-d3e760b52be7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.633222 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.633225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85sv9\" (UniqueName: \"kubernetes.io/projected/750732c0-2b7c-452a-9155-d3e760b52be7-kube-api-access-85sv9\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.633515 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.633249    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/750732c0-2b7c-452a-9155-d3e760b52be7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.633515 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.633294    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/750732c0-2b7c-452a-9155-d3e760b52be7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.733900 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.733852    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/750732c0-2b7c-452a-9155-d3e760b52be7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.733900 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.733906    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85sv9\" (UniqueName: \"kubernetes.io/projected/750732c0-2b7c-452a-9155-d3e760b52be7-kube-api-access-85sv9\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.734189 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.733981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/750732c0-2b7c-452a-9155-d3e760b52be7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.734189 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.734017    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/750732c0-2b7c-452a-9155-d3e760b52be7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.734484 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.734454    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/750732c0-2b7c-452a-9155-d3e760b52be7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.734744 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.734723    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/750732c0-2b7c-452a-9155-d3e760b52be7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.736659 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.736638    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/750732c0-2b7c-452a-9155-d3e760b52be7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.743192 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.743170    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85sv9\" (UniqueName: \"kubernetes.io/projected/750732c0-2b7c-452a-9155-d3e760b52be7-kube-api-access-85sv9\") pod \"isvc-sklearn-s3-predictor-7f677cd7-5wljh\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:05.815933 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.815840    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5325af82-01a1-423c-aaa2-396a35e1ccc1" path="/var/lib/kubelet/pods/5325af82-01a1-423c-aaa2-396a35e1ccc1/volumes"
Sep 16 19:29:05.875885 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:05.875839    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:06.180526 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:06.180501    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"]
Sep 16 19:29:06.182852 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:29:06.182824    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750732c0_2b7c_452a_9155_d3e760b52be7.slice/crio-449c9c2b7f28e9b07e8e99854beafa498d0cab797d2f9b1e6e5d7f6d0aa0e32e WatchSource:0}: Error finding container 449c9c2b7f28e9b07e8e99854beafa498d0cab797d2f9b1e6e5d7f6d0aa0e32e: Status 404 returned error can't find the container with id 449c9c2b7f28e9b07e8e99854beafa498d0cab797d2f9b1e6e5d7f6d0aa0e32e
Sep 16 19:29:06.279949 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:06.279910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" event={"ID":"750732c0-2b7c-452a-9155-d3e760b52be7","Type":"ContainerStarted","Data":"274c83e3deca593a259eb7e8378e6d5bbfc95981caeea1dfef6a1876cb177f4c"}
Sep 16 19:29:06.280168 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:06.279981    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" event={"ID":"750732c0-2b7c-452a-9155-d3e760b52be7","Type":"ContainerStarted","Data":"449c9c2b7f28e9b07e8e99854beafa498d0cab797d2f9b1e6e5d7f6d0aa0e32e"}
Sep 16 19:29:08.291485 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:08.291452    2584 generic.go:356] "Generic (PLEG): container finished" podID="750732c0-2b7c-452a-9155-d3e760b52be7" containerID="274c83e3deca593a259eb7e8378e6d5bbfc95981caeea1dfef6a1876cb177f4c" exitCode=0
Sep 16 19:29:08.291854 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:08.291523    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" event={"ID":"750732c0-2b7c-452a-9155-d3e760b52be7","Type":"ContainerDied","Data":"274c83e3deca593a259eb7e8378e6d5bbfc95981caeea1dfef6a1876cb177f4c"}
Sep 16 19:29:09.298004 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:09.297972    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" event={"ID":"750732c0-2b7c-452a-9155-d3e760b52be7","Type":"ContainerStarted","Data":"dc33148162e252ed92901b915fa2360927925e3958996ca9a11cc52cb25bc46f"}
Sep 16 19:29:09.298402 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:09.298065    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" event={"ID":"750732c0-2b7c-452a-9155-d3e760b52be7","Type":"ContainerStarted","Data":"d493e351f9cb26e742a48fde13b66da3a627082effb2b8124eecb3d581a4cfec"}
Sep 16 19:29:09.298402 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:09.298289    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:09.321467 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:09.321417    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podStartSLOduration=3.064835861 podStartE2EDuration="4.321401301s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.25656544s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:09.319923177 +0000 UTC m=+2679.986862351" watchObservedRunningTime="2026-09-16 19:29:09.321401301 +0000 UTC m=+2679.988340446"
Sep 16 19:29:10.303639 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:10.303606    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:10.305179 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:10.305150    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:29:11.308809 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:11.308763    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:29:16.315284 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:16.315242    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:29:16.315880 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:16.315857    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:29:26.316442 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:26.316348    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:29:36.315843 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:36.315801    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:29:46.316695 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:46.316653    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:29:56.316421 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:29:56.316381    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:30:06.315822 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:06.315782    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:30:16.316590 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:16.316545    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:30:16.811033 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:16.810988    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:30:26.812190 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:26.812154    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:30:35.773259 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:35.773223    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"]
Sep 16 19:30:35.773775 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:35.773542    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" containerID="cri-o://d493e351f9cb26e742a48fde13b66da3a627082effb2b8124eecb3d581a4cfec" gracePeriod=30
Sep 16 19:30:35.773775 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:35.773557    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kube-rbac-proxy" containerID="cri-o://dc33148162e252ed92901b915fa2360927925e3958996ca9a11cc52cb25bc46f" gracePeriod=30
Sep 16 19:30:35.927700 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:35.927658    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"]
Sep 16 19:30:35.931351 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:35.931318    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:35.934032 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:35.934012    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:30:35.934168 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:35.934151    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:30:35.934381 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:35.934364    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 16 19:30:35.944202 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:35.944162    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"]
Sep 16 19:30:36.086710 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.086621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.086710 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.086665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.086946 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.086792    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.086946 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.086830    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.086946 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.086860    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnn8p\" (UniqueName: \"kubernetes.io/projected/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-api-access-nnn8p\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.187996 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.187960    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.187996 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.188003    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.188261 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.188040    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.188261 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.188056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.188261 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.188075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnn8p\" (UniqueName: \"kubernetes.io/projected/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-api-access-nnn8p\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.188562 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.188539    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.188775 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.188753    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.188839 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.188780    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.190547 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.190527    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.197244 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.197223    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnn8p\" (UniqueName: \"kubernetes.io/projected/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-api-access-nnn8p\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.243319 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.243261    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:36.310380 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.309062    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 16 19:30:36.521982 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.521957    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"]
Sep 16 19:30:36.524170 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:30:36.524140    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e25ac75_9e4c_4d37_87b8_7d72b57006d5.slice/crio-c7b962fe852250262bb4d7733c59ac592b34787cdd87515a928ac776659d9d46 WatchSource:0}: Error finding container c7b962fe852250262bb4d7733c59ac592b34787cdd87515a928ac776659d9d46: Status 404 returned error can't find the container with id c7b962fe852250262bb4d7733c59ac592b34787cdd87515a928ac776659d9d46
Sep 16 19:30:36.706640 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.706598    2584 generic.go:356] "Generic (PLEG): container finished" podID="750732c0-2b7c-452a-9155-d3e760b52be7" containerID="dc33148162e252ed92901b915fa2360927925e3958996ca9a11cc52cb25bc46f" exitCode=2
Sep 16 19:30:36.706838 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.706672    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" event={"ID":"750732c0-2b7c-452a-9155-d3e760b52be7","Type":"ContainerDied","Data":"dc33148162e252ed92901b915fa2360927925e3958996ca9a11cc52cb25bc46f"}
Sep 16 19:30:36.708386 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.708359    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" event={"ID":"9e25ac75-9e4c-4d37-87b8-7d72b57006d5","Type":"ContainerStarted","Data":"4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1"}
Sep 16 19:30:36.708537 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.708408    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" event={"ID":"9e25ac75-9e4c-4d37-87b8-7d72b57006d5","Type":"ContainerStarted","Data":"c7b962fe852250262bb4d7733c59ac592b34787cdd87515a928ac776659d9d46"}
Sep 16 19:30:36.811807 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:36.811700    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 16 19:30:38.718149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:38.718116    2584 generic.go:356] "Generic (PLEG): container finished" podID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerID="4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1" exitCode=0
Sep 16 19:30:38.718551 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:38.718154    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" event={"ID":"9e25ac75-9e4c-4d37-87b8-7d72b57006d5","Type":"ContainerDied","Data":"4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1"}
Sep 16 19:30:39.724673 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:39.724639    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" event={"ID":"9e25ac75-9e4c-4d37-87b8-7d72b57006d5","Type":"ContainerStarted","Data":"0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa"}
Sep 16 19:30:39.725135 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:39.724729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" event={"ID":"9e25ac75-9e4c-4d37-87b8-7d72b57006d5","Type":"ContainerStarted","Data":"8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b"}
Sep 16 19:30:39.725135 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:39.725045    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:39.725242 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:39.725210    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:39.726583 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:39.726558    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:30:39.746729 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:39.746684    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podStartSLOduration=3.5845565969999997 podStartE2EDuration="4.746672011s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.162115414s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:39.745775913 +0000 UTC m=+2770.412715063" watchObservedRunningTime="2026-09-16 19:30:39.746672011 +0000 UTC m=+2770.413611148"
Sep 16 19:30:40.732324 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.732280    2584 generic.go:356] "Generic (PLEG): container finished" podID="750732c0-2b7c-452a-9155-d3e760b52be7" containerID="d493e351f9cb26e742a48fde13b66da3a627082effb2b8124eecb3d581a4cfec" exitCode=0
Sep 16 19:30:40.732324 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.732302    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" event={"ID":"750732c0-2b7c-452a-9155-d3e760b52be7","Type":"ContainerDied","Data":"d493e351f9cb26e742a48fde13b66da3a627082effb2b8124eecb3d581a4cfec"}
Sep 16 19:30:40.732877 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.732713    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:30:40.824658 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.824633    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:30:40.833450 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.833425    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/750732c0-2b7c-452a-9155-d3e760b52be7-proxy-tls\") pod \"750732c0-2b7c-452a-9155-d3e760b52be7\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") "
Sep 16 19:30:40.833577 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.833478    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85sv9\" (UniqueName: \"kubernetes.io/projected/750732c0-2b7c-452a-9155-d3e760b52be7-kube-api-access-85sv9\") pod \"750732c0-2b7c-452a-9155-d3e760b52be7\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") "
Sep 16 19:30:40.833577 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.833512    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/750732c0-2b7c-452a-9155-d3e760b52be7-kube-rbac-proxy-sar-config\") pod \"750732c0-2b7c-452a-9155-d3e760b52be7\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") "
Sep 16 19:30:40.833577 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.833564    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/750732c0-2b7c-452a-9155-d3e760b52be7-kserve-provision-location\") pod \"750732c0-2b7c-452a-9155-d3e760b52be7\" (UID: \"750732c0-2b7c-452a-9155-d3e760b52be7\") "
Sep 16 19:30:40.833944 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.833915    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750732c0-2b7c-452a-9155-d3e760b52be7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "750732c0-2b7c-452a-9155-d3e760b52be7" (UID: "750732c0-2b7c-452a-9155-d3e760b52be7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:40.833944 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.833934    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/750732c0-2b7c-452a-9155-d3e760b52be7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "750732c0-2b7c-452a-9155-d3e760b52be7" (UID: "750732c0-2b7c-452a-9155-d3e760b52be7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:40.835612 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.835592    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750732c0-2b7c-452a-9155-d3e760b52be7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "750732c0-2b7c-452a-9155-d3e760b52be7" (UID: "750732c0-2b7c-452a-9155-d3e760b52be7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:40.835689 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.835656    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750732c0-2b7c-452a-9155-d3e760b52be7-kube-api-access-85sv9" (OuterVolumeSpecName: "kube-api-access-85sv9") pod "750732c0-2b7c-452a-9155-d3e760b52be7" (UID: "750732c0-2b7c-452a-9155-d3e760b52be7"). InnerVolumeSpecName "kube-api-access-85sv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:40.934973 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.934935    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/750732c0-2b7c-452a-9155-d3e760b52be7-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:30:40.934973 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.934967    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/750732c0-2b7c-452a-9155-d3e760b52be7-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:30:40.934973 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.934977    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85sv9\" (UniqueName: \"kubernetes.io/projected/750732c0-2b7c-452a-9155-d3e760b52be7-kube-api-access-85sv9\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:30:40.935211 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:40.934987    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/750732c0-2b7c-452a-9155-d3e760b52be7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:30:41.738992 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:41.738962    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"
Sep 16 19:30:41.738992 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:41.738985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh" event={"ID":"750732c0-2b7c-452a-9155-d3e760b52be7","Type":"ContainerDied","Data":"449c9c2b7f28e9b07e8e99854beafa498d0cab797d2f9b1e6e5d7f6d0aa0e32e"}
Sep 16 19:30:41.739537 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:41.739046    2584 scope.go:117] "RemoveContainer" containerID="dc33148162e252ed92901b915fa2360927925e3958996ca9a11cc52cb25bc46f"
Sep 16 19:30:41.739590 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:41.739565    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:30:41.748332 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:41.748310    2584 scope.go:117] "RemoveContainer" containerID="d493e351f9cb26e742a48fde13b66da3a627082effb2b8124eecb3d581a4cfec"
Sep 16 19:30:41.756444 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:41.756426    2584 scope.go:117] "RemoveContainer" containerID="274c83e3deca593a259eb7e8378e6d5bbfc95981caeea1dfef6a1876cb177f4c"
Sep 16 19:30:41.762399 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:41.762374    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"]
Sep 16 19:30:41.770317 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:41.770288    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7f677cd7-5wljh"]
Sep 16 19:30:41.817979 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:41.817947    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" path="/var/lib/kubelet/pods/750732c0-2b7c-452a-9155-d3e760b52be7/volumes"
Sep 16 19:30:46.744232 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:46.744202    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:30:46.744796 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:46.744768    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:30:56.744840 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:30:56.744800    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:31:06.745355 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:31:06.745316    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:31:16.745014 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:31:16.744972    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:31:26.745064 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:31:26.745024    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:31:36.744932 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:31:36.744895    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:31:46.745685 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:31:46.745647    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:31:56.745903 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:31:56.745871    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:32:05.910165 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:05.910130    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"]
Sep 16 19:32:05.910567 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:05.910461    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" containerID="cri-o://8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b" gracePeriod=30
Sep 16 19:32:05.910567 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:05.910488    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kube-rbac-proxy" containerID="cri-o://0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa" gracePeriod=30
Sep 16 19:32:06.141366 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:06.141325    2584 generic.go:356] "Generic (PLEG): container finished" podID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerID="0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa" exitCode=2
Sep 16 19:32:06.141546 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:06.141412    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" event={"ID":"9e25ac75-9e4c-4d37-87b8-7d72b57006d5","Type":"ContainerDied","Data":"0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa"}
Sep 16 19:32:06.740061 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:06.740021    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Sep 16 19:32:06.744662 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:06.744641    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 16 19:32:10.486952 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.486928    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:32:10.540390 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.540364    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnn8p\" (UniqueName: \"kubernetes.io/projected/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-api-access-nnn8p\") pod \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") "
Sep 16 19:32:10.540572 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.540414    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kserve-provision-location\") pod \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") "
Sep 16 19:32:10.540572 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.540435    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-rbac-proxy-sar-config\") pod \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") "
Sep 16 19:32:10.540572 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.540475    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-cabundle-cert\") pod \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") "
Sep 16 19:32:10.540572 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.540518    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-proxy-tls\") pod \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\" (UID: \"9e25ac75-9e4c-4d37-87b8-7d72b57006d5\") "
Sep 16 19:32:10.540800 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.540718    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e25ac75-9e4c-4d37-87b8-7d72b57006d5" (UID: "9e25ac75-9e4c-4d37-87b8-7d72b57006d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:10.540881 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.540839    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e25ac75-9e4c-4d37-87b8-7d72b57006d5" (UID: "9e25ac75-9e4c-4d37-87b8-7d72b57006d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:10.540935 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.540862    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9e25ac75-9e4c-4d37-87b8-7d72b57006d5" (UID: "9e25ac75-9e4c-4d37-87b8-7d72b57006d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:10.542574 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.542553    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-api-access-nnn8p" (OuterVolumeSpecName: "kube-api-access-nnn8p") pod "9e25ac75-9e4c-4d37-87b8-7d72b57006d5" (UID: "9e25ac75-9e4c-4d37-87b8-7d72b57006d5"). InnerVolumeSpecName "kube-api-access-nnn8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:10.542574 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.542560    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e25ac75-9e4c-4d37-87b8-7d72b57006d5" (UID: "9e25ac75-9e4c-4d37-87b8-7d72b57006d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:10.642022 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.641942    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:32:10.642022 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.641970    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnn8p\" (UniqueName: \"kubernetes.io/projected/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-api-access-nnn8p\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:32:10.642022 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.641981    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:32:10.642022 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.641990    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:32:10.642022 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:10.641999    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e25ac75-9e4c-4d37-87b8-7d72b57006d5-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:32:11.166533 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.166495    2584 generic.go:356] "Generic (PLEG): container finished" podID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerID="8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b" exitCode=0
Sep 16 19:32:11.166707 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.166551    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"
Sep 16 19:32:11.166707 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.166579    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" event={"ID":"9e25ac75-9e4c-4d37-87b8-7d72b57006d5","Type":"ContainerDied","Data":"8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b"}
Sep 16 19:32:11.166707 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.166631    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s" event={"ID":"9e25ac75-9e4c-4d37-87b8-7d72b57006d5","Type":"ContainerDied","Data":"c7b962fe852250262bb4d7733c59ac592b34787cdd87515a928ac776659d9d46"}
Sep 16 19:32:11.166707 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.166660    2584 scope.go:117] "RemoveContainer" containerID="0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa"
Sep 16 19:32:11.176924 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.176904    2584 scope.go:117] "RemoveContainer" containerID="8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b"
Sep 16 19:32:11.184819 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.184804    2584 scope.go:117] "RemoveContainer" containerID="4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1"
Sep 16 19:32:11.192317 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.192285    2584 scope.go:117] "RemoveContainer" containerID="0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa"
Sep 16 19:32:11.192650 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:32:11.192630    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa\": container with ID starting with 0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa not found: ID does not exist" containerID="0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa"
Sep 16 19:32:11.192698 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.192659    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa"} err="failed to get container status \"0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa\": rpc error: code = NotFound desc = could not find container \"0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa\": container with ID starting with 0a2b7e63fb91c9da3ff582d9acef7b85299eabfec55899c8f94c069a2ec1cafa not found: ID does not exist"
Sep 16 19:32:11.192698 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.192676    2584 scope.go:117] "RemoveContainer" containerID="8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b"
Sep 16 19:32:11.192925 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:32:11.192907    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b\": container with ID starting with 8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b not found: ID does not exist" containerID="8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b"
Sep 16 19:32:11.192970 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.192931    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b"} err="failed to get container status \"8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b\": rpc error: code = NotFound desc = could not find container \"8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b\": container with ID starting with 8a04553cb12102a29a460ba7c5041eddd868f43d8558fe94fedde9f27702944b not found: ID does not exist"
Sep 16 19:32:11.192970 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.192947    2584 scope.go:117] "RemoveContainer" containerID="4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1"
Sep 16 19:32:11.193049 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.192946    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"]
Sep 16 19:32:11.193231 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:32:11.193214    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1\": container with ID starting with 4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1 not found: ID does not exist" containerID="4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1"
Sep 16 19:32:11.193347 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.193239    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1"} err="failed to get container status \"4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1\": rpc error: code = NotFound desc = could not find container \"4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1\": container with ID starting with 4b741c17ef667a9433f45ded583ee46c8ea02c26e2dead16b4fe2c668b3befb1 not found: ID does not exist"
Sep 16 19:32:11.196748 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.196725    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-855c6b599-g5n4s"]
Sep 16 19:32:11.816651 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:11.816618    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" path="/var/lib/kubelet/pods/9e25ac75-9e4c-4d37-87b8-7d72b57006d5/volumes"
Sep 16 19:32:18.185887 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.185851    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"]
Sep 16 19:32:18.186425 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186409    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kube-rbac-proxy"
Sep 16 19:32:18.186481 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186428    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kube-rbac-proxy"
Sep 16 19:32:18.186481 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186449    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="storage-initializer"
Sep 16 19:32:18.186481 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186458    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="storage-initializer"
Sep 16 19:32:18.186481 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186475    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kube-rbac-proxy"
Sep 16 19:32:18.186610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186484    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kube-rbac-proxy"
Sep 16 19:32:18.186610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186501    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="storage-initializer"
Sep 16 19:32:18.186610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186510    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="storage-initializer"
Sep 16 19:32:18.186610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186522    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container"
Sep 16 19:32:18.186610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186531    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container"
Sep 16 19:32:18.186610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186545    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container"
Sep 16 19:32:18.186610 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186553    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container"
Sep 16 19:32:18.186819 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186629    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kube-rbac-proxy"
Sep 16 19:32:18.186819 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186637    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kserve-container"
Sep 16 19:32:18.186819 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186649    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="750732c0-2b7c-452a-9155-d3e760b52be7" containerName="kserve-container"
Sep 16 19:32:18.186819 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.186658    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e25ac75-9e4c-4d37-87b8-7d72b57006d5" containerName="kube-rbac-proxy"
Sep 16 19:32:18.191902 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.191878    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.195093 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.195069    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:32:18.195240 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.195117    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:32:18.195240 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.195211    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:32:18.196315 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.196296    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 16 19:32:18.196315 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.196303    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:32:18.196500 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.196330    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:32:18.196500 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.196372    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:32:18.206202 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.206176    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"]
Sep 16 19:32:18.304035 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.303998    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.304035 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.304040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.304240 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.304082    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv949\" (UniqueName: \"kubernetes.io/projected/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-api-access-jv949\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.304240 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.304101    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.304240 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.304133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.404798 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.404743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.404798 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.404803    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.405044 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.404837    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv949\" (UniqueName: \"kubernetes.io/projected/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-api-access-jv949\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.405044 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.404857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.405044 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.404883    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.405432 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.405408    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.405509 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.405490    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.405592 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.405573    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.407541 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.407522    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.416045 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.416012    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv949\" (UniqueName: \"kubernetes.io/projected/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-api-access-jv949\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.502581 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.502482    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:18.785036 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:18.785011    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"]
Sep 16 19:32:18.787083 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:32:18.787054    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb2745f_0cee_4407_8b7e_e8e5db5897c7.slice/crio-099333c4b47979c50c3730b83470db48ca36afe3be22a7863454d6a35523162c WatchSource:0}: Error finding container 099333c4b47979c50c3730b83470db48ca36afe3be22a7863454d6a35523162c: Status 404 returned error can't find the container with id 099333c4b47979c50c3730b83470db48ca36afe3be22a7863454d6a35523162c
Sep 16 19:32:19.205969 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:19.205927    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" event={"ID":"1eb2745f-0cee-4407-8b7e-e8e5db5897c7","Type":"ContainerStarted","Data":"8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6"}
Sep 16 19:32:19.206396 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:19.206005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" event={"ID":"1eb2745f-0cee-4407-8b7e-e8e5db5897c7","Type":"ContainerStarted","Data":"099333c4b47979c50c3730b83470db48ca36afe3be22a7863454d6a35523162c"}
Sep 16 19:32:20.212312 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:20.212259    2584 generic.go:356] "Generic (PLEG): container finished" podID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerID="8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6" exitCode=0
Sep 16 19:32:20.212691 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:20.212355    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" event={"ID":"1eb2745f-0cee-4407-8b7e-e8e5db5897c7","Type":"ContainerDied","Data":"8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6"}
Sep 16 19:32:21.219761 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:21.219727    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" event={"ID":"1eb2745f-0cee-4407-8b7e-e8e5db5897c7","Type":"ContainerStarted","Data":"ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8"}
Sep 16 19:32:21.220206 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:21.219820    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" event={"ID":"1eb2745f-0cee-4407-8b7e-e8e5db5897c7","Type":"ContainerStarted","Data":"2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5"}
Sep 16 19:32:21.220206 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:21.219966    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:21.220206 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:21.220021    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:21.221540 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:21.221514    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:32:21.241317 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:21.241251    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podStartSLOduration=2.120546162 podStartE2EDuration="3.241235965s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.120689803s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:21.239169524 +0000 UTC m=+2871.906108692" watchObservedRunningTime="2026-09-16 19:32:21.241235965 +0000 UTC m=+2871.908175111"
Sep 16 19:32:22.225323 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:22.225221    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:32:24.300867 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.300828    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"]
Sep 16 19:32:24.304310 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.304282    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.307230 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.307209    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 16 19:32:24.307378 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.307216    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:32:24.307378 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.307249    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:24.316171 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.316147    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"]
Sep 16 19:32:24.357347 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.357310    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.357497 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.357365    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.357497 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.357398    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.357497 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.357428    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcp6w\" (UniqueName: \"kubernetes.io/projected/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-api-access-gcp6w\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.357497 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.357488    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.458350 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.458313    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.458350 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.458351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcp6w\" (UniqueName: \"kubernetes.io/projected/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-api-access-gcp6w\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.458571 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.458391    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.458571 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.458437    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.458571 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.458519    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.458571 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:32:24.458568    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 16 19:32:24.458775 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:32:24.458642    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-proxy-tls podName:13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5 nodeName:}" failed. No retries permitted until 2026-09-16 19:32:24.958620189 +0000 UTC m=+2875.625559338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-crj8v" (UID: "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 16 19:32:24.458942 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.458920    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.459033 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.459015    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.459115 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.459099    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.470188 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.470162    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcp6w\" (UniqueName: \"kubernetes.io/projected/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-api-access-gcp6w\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.963163 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.963120    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:24.965553 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:24.965532    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-crj8v\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:25.217090 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:25.216991    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:25.491898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:25.491717    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"]
Sep 16 19:32:25.494769 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:32:25.494739    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a46a0a_7a62_47c1_bce8_4a2ee1d5e8c5.slice/crio-270b1135c99530cf99fea04cb6e63a96d3b248d8493cbb5eb2c09aaeea435e37 WatchSource:0}: Error finding container 270b1135c99530cf99fea04cb6e63a96d3b248d8493cbb5eb2c09aaeea435e37: Status 404 returned error can't find the container with id 270b1135c99530cf99fea04cb6e63a96d3b248d8493cbb5eb2c09aaeea435e37
Sep 16 19:32:26.245362 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:26.245326    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" event={"ID":"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5","Type":"ContainerStarted","Data":"fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0"}
Sep 16 19:32:26.245546 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:26.245396    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" event={"ID":"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5","Type":"ContainerStarted","Data":"270b1135c99530cf99fea04cb6e63a96d3b248d8493cbb5eb2c09aaeea435e37"}
Sep 16 19:32:27.235766 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:27.235732    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:32:27.236395 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:27.236364    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:32:30.266665 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:30.266574    2584 generic.go:356] "Generic (PLEG): container finished" podID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerID="fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0" exitCode=0
Sep 16 19:32:30.267163 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:30.266652    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" event={"ID":"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5","Type":"ContainerDied","Data":"fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0"}
Sep 16 19:32:30.268003 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:30.267989    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:32:37.236719 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:37.236676    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:32:41.332481 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:41.332442    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" event={"ID":"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5","Type":"ContainerStarted","Data":"eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07"}
Sep 16 19:32:42.338669 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:42.338640    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" event={"ID":"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5","Type":"ContainerStarted","Data":"1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69"}
Sep 16 19:32:42.339045 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:42.338795    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:42.362785 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:42.362736    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" podStartSLOduration=2.9780355099999998 podStartE2EDuration="18.362719954s" totalImagesPullingTime="10.988260701s" totalInitContainerRuntime="4.396423743s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:42.360474011 +0000 UTC m=+2893.027413172" watchObservedRunningTime="2026-09-16 19:32:42.362719954 +0000 UTC m=+2893.029659099"
Sep 16 19:32:43.343411 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:43.343379    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:43.344497 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:43.344473    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 16 19:32:44.348308 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:44.348242    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 16 19:32:47.236699 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:47.236658    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:32:49.352764 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:49.352731    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:32:49.353292 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:49.353208    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 16 19:32:57.236665 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:57.236625    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:32:59.353584 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:32:59.353546    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 16 19:33:07.236511 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:07.236466    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:33:09.353485 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:09.353445    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 16 19:33:17.236722 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:17.236677    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:33:19.354224 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:19.354183    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 16 19:33:27.236885 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:27.236841    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:33:29.354181 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:29.354149    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:33:32.811714 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:32.811675    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:33:35.350329 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:35.350288    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"]
Sep 16 19:33:35.350795 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:35.350582    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container" containerID="cri-o://eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07" gracePeriod=30
Sep 16 19:33:35.350795 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:35.350691    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kube-rbac-proxy" containerID="cri-o://1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69" gracePeriod=30
Sep 16 19:33:35.589526 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:35.589487    2584 generic.go:356] "Generic (PLEG): container finished" podID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerID="1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69" exitCode=2
Sep 16 19:33:35.589697 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:35.589558    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" event={"ID":"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5","Type":"ContainerDied","Data":"1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69"}
Sep 16 19:33:38.302185 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.302157    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:33:38.305934 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.305911    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-cabundle-cert\") pod \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") "
Sep 16 19:33:38.306121 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.305940    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-proxy-tls\") pod \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") "
Sep 16 19:33:38.306121 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.305975    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kserve-provision-location\") pod \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") "
Sep 16 19:33:38.306121 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.306093    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-rbac-proxy-sar-config\") pod \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") "
Sep 16 19:33:38.306362 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.306341    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" (UID: "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:38.306446 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.306423    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" (UID: "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:38.308048 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.308023    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" (UID: "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:38.315765 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.315738    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" (UID: "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:38.407003 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.406906    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcp6w\" (UniqueName: \"kubernetes.io/projected/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-api-access-gcp6w\") pod \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\" (UID: \"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5\") "
Sep 16 19:33:38.407149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.407085    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:38.407149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.407097    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:38.407149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.407106    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:38.407149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.407115    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:38.409025 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.409000    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-api-access-gcp6w" (OuterVolumeSpecName: "kube-api-access-gcp6w") pod "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" (UID: "13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5"). InnerVolumeSpecName "kube-api-access-gcp6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:38.507816 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.507779    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcp6w\" (UniqueName: \"kubernetes.io/projected/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5-kube-api-access-gcp6w\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:38.605559 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.605521    2584 generic.go:356] "Generic (PLEG): container finished" podID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerID="eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07" exitCode=0
Sep 16 19:33:38.605728 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.605583    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"
Sep 16 19:33:38.605728 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.605591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" event={"ID":"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5","Type":"ContainerDied","Data":"eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07"}
Sep 16 19:33:38.605728 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.605652    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v" event={"ID":"13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5","Type":"ContainerDied","Data":"270b1135c99530cf99fea04cb6e63a96d3b248d8493cbb5eb2c09aaeea435e37"}
Sep 16 19:33:38.605728 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.605688    2584 scope.go:117] "RemoveContainer" containerID="1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69"
Sep 16 19:33:38.614822 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.614800    2584 scope.go:117] "RemoveContainer" containerID="eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07"
Sep 16 19:33:38.622699 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.622680    2584 scope.go:117] "RemoveContainer" containerID="fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0"
Sep 16 19:33:38.628855 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.628833    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"]
Sep 16 19:33:38.631278 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.631244    2584 scope.go:117] "RemoveContainer" containerID="1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69"
Sep 16 19:33:38.631655 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:33:38.631625    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69\": container with ID starting with 1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69 not found: ID does not exist" containerID="1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69"
Sep 16 19:33:38.631747 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.631661    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69"} err="failed to get container status \"1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69\": rpc error: code = NotFound desc = could not find container \"1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69\": container with ID starting with 1b5a8213ba42901dae703c9da2e058d8ba0f920295e193413057e5e15c64fb69 not found: ID does not exist"
Sep 16 19:33:38.631747 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.631680    2584 scope.go:117] "RemoveContainer" containerID="eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07"
Sep 16 19:33:38.631974 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:33:38.631958    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07\": container with ID starting with eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07 not found: ID does not exist" containerID="eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07"
Sep 16 19:33:38.632026 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.631978    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07"} err="failed to get container status \"eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07\": rpc error: code = NotFound desc = could not find container \"eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07\": container with ID starting with eb05bf3ddab75102fc3f0e8edab0021feb47a53cd70e44dd05830d050f8a6f07 not found: ID does not exist"
Sep 16 19:33:38.632026 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.631990    2584 scope.go:117] "RemoveContainer" containerID="fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0"
Sep 16 19:33:38.632223 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:33:38.632207    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0\": container with ID starting with fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0 not found: ID does not exist" containerID="fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0"
Sep 16 19:33:38.632301 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.632229    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0"} err="failed to get container status \"fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0\": rpc error: code = NotFound desc = could not find container \"fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0\": container with ID starting with fde252c84989e151e11d18a59c6a0b912f47622fb7a2f37d2d1c123fcd45b5a0 not found: ID does not exist"
Sep 16 19:33:38.632562 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:38.632543    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-crj8v"]
Sep 16 19:33:39.816703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:39.816668    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" path="/var/lib/kubelet/pods/13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5/volumes"
Sep 16 19:33:42.812451 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:42.812411    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:33:48.194393 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:48.194362    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"]
Sep 16 19:33:48.194800 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:48.194584    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" containerID="cri-o://2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5" gracePeriod=30
Sep 16 19:33:48.194800 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:48.194614    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kube-rbac-proxy" containerID="cri-o://ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8" gracePeriod=30
Sep 16 19:33:48.654350 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:48.654250    2584 generic.go:356] "Generic (PLEG): container finished" podID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerID="ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8" exitCode=2
Sep 16 19:33:48.654350 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:48.654292    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" event={"ID":"1eb2745f-0cee-4407-8b7e-e8e5db5897c7","Type":"ContainerDied","Data":"ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8"}
Sep 16 19:33:52.226489 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:52.226392    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Sep 16 19:33:52.811670 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:52.811623    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 16 19:33:53.341514 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.341491    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:33:53.434125 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.434095    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-proxy-tls\") pod \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") "
Sep 16 19:33:53.434339 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.434145    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-rbac-proxy-sar-config\") pod \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") "
Sep 16 19:33:53.434339 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.434163    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-cabundle-cert\") pod \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") "
Sep 16 19:33:53.434339 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.434188    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kserve-provision-location\") pod \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") "
Sep 16 19:33:53.434339 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.434244    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv949\" (UniqueName: \"kubernetes.io/projected/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-api-access-jv949\") pod \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\" (UID: \"1eb2745f-0cee-4407-8b7e-e8e5db5897c7\") "
Sep 16 19:33:53.434573 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.434531    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1eb2745f-0cee-4407-8b7e-e8e5db5897c7" (UID: "1eb2745f-0cee-4407-8b7e-e8e5db5897c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:53.434653 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.434578    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1eb2745f-0cee-4407-8b7e-e8e5db5897c7" (UID: "1eb2745f-0cee-4407-8b7e-e8e5db5897c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:53.434723 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.434695    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1eb2745f-0cee-4407-8b7e-e8e5db5897c7" (UID: "1eb2745f-0cee-4407-8b7e-e8e5db5897c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:53.436322 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.436300    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1eb2745f-0cee-4407-8b7e-e8e5db5897c7" (UID: "1eb2745f-0cee-4407-8b7e-e8e5db5897c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:53.436409 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.436340    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-api-access-jv949" (OuterVolumeSpecName: "kube-api-access-jv949") pod "1eb2745f-0cee-4407-8b7e-e8e5db5897c7" (UID: "1eb2745f-0cee-4407-8b7e-e8e5db5897c7"). InnerVolumeSpecName "kube-api-access-jv949". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:53.535603 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.535570    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:53.535603 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.535601    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:53.535603 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.535612    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:53.535834 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.535621    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv949\" (UniqueName: \"kubernetes.io/projected/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-kube-api-access-jv949\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:53.535834 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.535631    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eb2745f-0cee-4407-8b7e-e8e5db5897c7-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:33:53.678480 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.678448    2584 generic.go:356] "Generic (PLEG): container finished" podID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerID="2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5" exitCode=0
Sep 16 19:33:53.678625 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.678492    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" event={"ID":"1eb2745f-0cee-4407-8b7e-e8e5db5897c7","Type":"ContainerDied","Data":"2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5"}
Sep 16 19:33:53.678625 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.678509    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"
Sep 16 19:33:53.678625 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.678537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr" event={"ID":"1eb2745f-0cee-4407-8b7e-e8e5db5897c7","Type":"ContainerDied","Data":"099333c4b47979c50c3730b83470db48ca36afe3be22a7863454d6a35523162c"}
Sep 16 19:33:53.678625 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.678565    2584 scope.go:117] "RemoveContainer" containerID="ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8"
Sep 16 19:33:53.688406 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.688379    2584 scope.go:117] "RemoveContainer" containerID="2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5"
Sep 16 19:33:53.698004 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.697978    2584 scope.go:117] "RemoveContainer" containerID="8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6"
Sep 16 19:33:53.703087 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.703057    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"]
Sep 16 19:33:53.706352 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.706320    2584 scope.go:117] "RemoveContainer" containerID="ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8"
Sep 16 19:33:53.706809 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:33:53.706716    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8\": container with ID starting with ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8 not found: ID does not exist" containerID="ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8"
Sep 16 19:33:53.706809 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.706755    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8"} err="failed to get container status \"ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8\": rpc error: code = NotFound desc = could not find container \"ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8\": container with ID starting with ee3dea6abea807b2644e21d28f3fea1e851801125d1f7d94fe85a2b6eba222a8 not found: ID does not exist"
Sep 16 19:33:53.706809 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.706780    2584 scope.go:117] "RemoveContainer" containerID="2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5"
Sep 16 19:33:53.707102 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:33:53.707085    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5\": container with ID starting with 2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5 not found: ID does not exist" containerID="2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5"
Sep 16 19:33:53.707167 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.707109    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5"} err="failed to get container status \"2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5\": rpc error: code = NotFound desc = could not find container \"2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5\": container with ID starting with 2f791baca5e9a16ae42c0f76e17cd3f3220d00edc9845ab4892120470cf357b5 not found: ID does not exist"
Sep 16 19:33:53.707167 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.707130    2584 scope.go:117] "RemoveContainer" containerID="8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6"
Sep 16 19:33:53.707499 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:33:53.707480    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6\": container with ID starting with 8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6 not found: ID does not exist" containerID="8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6"
Sep 16 19:33:53.707575 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.707504    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6"} err="failed to get container status \"8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6\": rpc error: code = NotFound desc = could not find container \"8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6\": container with ID starting with 8f28b6d1a91fa44b4f7d1992e0ffb0ee5015340259e1aef1d0a71be148e31fd6 not found: ID does not exist"
Sep 16 19:33:53.708499 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.708480    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-65d845549f-22ppr"]
Sep 16 19:33:53.815811 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:33:53.815780    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" path="/var/lib/kubelet/pods/1eb2745f-0cee-4407-8b7e-e8e5db5897c7/volumes"
Sep 16 19:34:00.417307 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417256    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"]
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417630    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417644    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417655    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kube-rbac-proxy"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417661    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kube-rbac-proxy"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417673    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417679    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417687    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kube-rbac-proxy"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417691    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kube-rbac-proxy"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417699    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="storage-initializer"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417706    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="storage-initializer"
Sep 16 19:34:00.417713 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417716    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="storage-initializer"
Sep 16 19:34:00.418170 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417724    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="storage-initializer"
Sep 16 19:34:00.418170 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417790    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kube-rbac-proxy"
Sep 16 19:34:00.418170 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417806    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a46a0a-7a62-47c1-bce8-4a2ee1d5e8c5" containerName="kserve-container"
Sep 16 19:34:00.418170 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417813    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kserve-container"
Sep 16 19:34:00.418170 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.417820    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb2745f-0cee-4407-8b7e-e8e5db5897c7" containerName="kube-rbac-proxy"
Sep 16 19:34:00.422600 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.422572    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.425507 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.425478    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:34:00.425677 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.425485    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:34:00.425677 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.425549    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 16 19:34:00.425821 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.425687    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:34:00.425821 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.425687    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:34:00.425821 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.425711    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:34:00.426298 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.426277    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:34:00.432066 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.432042    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"]
Sep 16 19:34:00.495939 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.495901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.496118 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.495946    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzbm\" (UniqueName: \"kubernetes.io/projected/d2a81649-573f-4427-891d-050cd947ac64-kube-api-access-wmzbm\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.496118 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.495969    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.496118 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.496042    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.496118 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.496072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a81649-573f-4427-891d-050cd947ac64-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.596675 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.596629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.596675 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.596680    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzbm\" (UniqueName: \"kubernetes.io/projected/d2a81649-573f-4427-891d-050cd947ac64-kube-api-access-wmzbm\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.596940 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.596817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.596940 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.596874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.596940 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.596933    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a81649-573f-4427-891d-050cd947ac64-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.597099 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:34:00.597059    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 16 19:34:00.597150 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:34:00.597142    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls podName:d2a81649-573f-4427-891d-050cd947ac64 nodeName:}" failed. No retries permitted until 2026-09-16 19:34:01.097120366 +0000 UTC m=+2971.764059491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" (UID: "d2a81649-573f-4427-891d-050cd947ac64") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 16 19:34:00.597359 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.597340    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a81649-573f-4427-891d-050cd947ac64-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.597446 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.597429    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.597488 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.597435    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:00.605341 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:00.605315    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzbm\" (UniqueName: \"kubernetes.io/projected/d2a81649-573f-4427-891d-050cd947ac64-kube-api-access-wmzbm\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:01.101961 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:01.101926    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:01.102153 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:34:01.102049    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 16 19:34:01.102153 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:34:01.102117    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls podName:d2a81649-573f-4427-891d-050cd947ac64 nodeName:}" failed. No retries permitted until 2026-09-16 19:34:02.102099827 +0000 UTC m=+2972.769038958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" (UID: "d2a81649-573f-4427-891d-050cd947ac64") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 16 19:34:02.109899 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:02.109857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:02.112426 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:02.112401    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:02.235453 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:02.235411    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:02.512365 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:02.512337    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"]
Sep 16 19:34:02.514874 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:34:02.514844    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a81649_573f_4427_891d_050cd947ac64.slice/crio-571d0bc9995c0b4d8ebc9b0d7e0ec899f5bf8cb3443f027a3180b47ac8fc4aa8 WatchSource:0}: Error finding container 571d0bc9995c0b4d8ebc9b0d7e0ec899f5bf8cb3443f027a3180b47ac8fc4aa8: Status 404 returned error can't find the container with id 571d0bc9995c0b4d8ebc9b0d7e0ec899f5bf8cb3443f027a3180b47ac8fc4aa8
Sep 16 19:34:02.718811 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:02.718777    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" event={"ID":"d2a81649-573f-4427-891d-050cd947ac64","Type":"ContainerStarted","Data":"fc525f4856b575f1daa7d7be301a176f6b0b0df96d91bd182d36ec708c01d798"}
Sep 16 19:34:02.718973 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:02.718832    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" event={"ID":"d2a81649-573f-4427-891d-050cd947ac64","Type":"ContainerStarted","Data":"571d0bc9995c0b4d8ebc9b0d7e0ec899f5bf8cb3443f027a3180b47ac8fc4aa8"}
Sep 16 19:34:03.725424 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:03.725394    2584 generic.go:356] "Generic (PLEG): container finished" podID="d2a81649-573f-4427-891d-050cd947ac64" containerID="fc525f4856b575f1daa7d7be301a176f6b0b0df96d91bd182d36ec708c01d798" exitCode=0
Sep 16 19:34:03.725786 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:03.725458    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" event={"ID":"d2a81649-573f-4427-891d-050cd947ac64","Type":"ContainerDied","Data":"fc525f4856b575f1daa7d7be301a176f6b0b0df96d91bd182d36ec708c01d798"}
Sep 16 19:34:04.732643 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:04.732608    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" event={"ID":"d2a81649-573f-4427-891d-050cd947ac64","Type":"ContainerStarted","Data":"ce2a0655b77c7223c22a8318069cd7b70ab752ec9d19e205bf8c9e88dc4e40a6"}
Sep 16 19:34:04.733042 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:04.732664    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" event={"ID":"d2a81649-573f-4427-891d-050cd947ac64","Type":"ContainerStarted","Data":"51245a9f291a8cd1bc42ad33878bafd69feb0edbcfed944de9f0de7cb55ea97d"}
Sep 16 19:34:04.733042 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:04.732836    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:04.755526 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:04.755476    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podStartSLOduration=3.674864814 podStartE2EDuration="4.755461556s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.080596742s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:04.75279704 +0000 UTC m=+2975.419736207" watchObservedRunningTime="2026-09-16 19:34:04.755461556 +0000 UTC m=+2975.422400703"
Sep 16 19:34:05.737094 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:05.737061    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:05.738528 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:05.738491    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:34:06.741703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:06.741662    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:34:11.747399 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:11.747370    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:34:11.748043 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:11.748013    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:34:21.748088 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:21.748047    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:34:31.748705 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:31.748663    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:34:41.748369 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:41.748329    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:34:47.121891 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.121858    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"]
Sep 16 19:34:47.125496 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.125479    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.130655 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.130623    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 16 19:34:47.130789 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.130629    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:34:47.130789 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.130674    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 16 19:34:47.150815 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.150791    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"]
Sep 16 19:34:47.160171 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.160133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrrm\" (UniqueName: \"kubernetes.io/projected/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-api-access-zhrrm\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.160304 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.160194    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.160304 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.160243    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.160426 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.160329    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.160426 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.160364    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.261483 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.261451    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrrm\" (UniqueName: \"kubernetes.io/projected/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-api-access-zhrrm\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.261676 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.261499    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.261676 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.261520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.261676 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.261561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.261676 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.261583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.262096 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.262048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.262343 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.262318    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.262462 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.262407    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.264150 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.264127    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.272519 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.272498    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrrm\" (UniqueName: \"kubernetes.io/projected/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-api-access-zhrrm\") pod \"isvc-pmml-predictor-8674795f59-9dbh6\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.435939 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.435908    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:34:47.710813 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.710791    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"]
Sep 16 19:34:47.712465 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:34:47.712432    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e79c9c_1f5b_4206_966b_f15b02d8ebfc.slice/crio-3c6ea63c0494a33248edafedbeacaf22181cf241b1055a4a149aed57ab60fdff WatchSource:0}: Error finding container 3c6ea63c0494a33248edafedbeacaf22181cf241b1055a4a149aed57ab60fdff: Status 404 returned error can't find the container with id 3c6ea63c0494a33248edafedbeacaf22181cf241b1055a4a149aed57ab60fdff
Sep 16 19:34:47.929427 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.929389    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" event={"ID":"26e79c9c-1f5b-4206-966b-f15b02d8ebfc","Type":"ContainerStarted","Data":"91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663"}
Sep 16 19:34:47.929625 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:47.929457    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" event={"ID":"26e79c9c-1f5b-4206-966b-f15b02d8ebfc","Type":"ContainerStarted","Data":"3c6ea63c0494a33248edafedbeacaf22181cf241b1055a4a149aed57ab60fdff"}
Sep 16 19:34:51.748046 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:51.748003    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:34:51.949467 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:51.949435    2584 generic.go:356] "Generic (PLEG): container finished" podID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerID="91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663" exitCode=0
Sep 16 19:34:51.949638 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:51.949484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" event={"ID":"26e79c9c-1f5b-4206-966b-f15b02d8ebfc","Type":"ContainerDied","Data":"91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663"}
Sep 16 19:34:59.997729 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:59.997690    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" event={"ID":"26e79c9c-1f5b-4206-966b-f15b02d8ebfc","Type":"ContainerStarted","Data":"ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38"}
Sep 16 19:34:59.998120 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:59.997794    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" event={"ID":"26e79c9c-1f5b-4206-966b-f15b02d8ebfc","Type":"ContainerStarted","Data":"6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac"}
Sep 16 19:34:59.998120 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:34:59.997979    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:35:00.019550 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:00.019493    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podStartSLOduration=1.95680223 podStartE2EDuration="13.019475287s" totalImagesPullingTime="7.071547068s" totalInitContainerRuntime="3.991125989s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:00.017565576 +0000 UTC m=+3030.684504768" watchObservedRunningTime="2026-09-16 19:35:00.019475287 +0000 UTC m=+3030.686414432"
Sep 16 19:35:01.002124 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:01.002090    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:35:01.003628 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:01.003600    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:35:01.748343 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:01.748299    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:35:02.008120 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:02.008020    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:35:07.012987 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:07.012954    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:35:07.013590 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:07.013563    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:35:11.748701 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:11.748659    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:35:14.810825 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:14.810776    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:35:17.013498 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:17.013451    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:35:24.811485 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:24.811398    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:35:27.013945 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:27.013908    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:35:30.444321 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:30.444289    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"]
Sep 16 19:35:30.444703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:30.444525    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" containerID="cri-o://51245a9f291a8cd1bc42ad33878bafd69feb0edbcfed944de9f0de7cb55ea97d" gracePeriod=30
Sep 16 19:35:30.444703 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:30.444625    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kube-rbac-proxy" containerID="cri-o://ce2a0655b77c7223c22a8318069cd7b70ab752ec9d19e205bf8c9e88dc4e40a6" gracePeriod=30
Sep 16 19:35:31.144106 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:31.144073    2584 generic.go:356] "Generic (PLEG): container finished" podID="d2a81649-573f-4427-891d-050cd947ac64" containerID="ce2a0655b77c7223c22a8318069cd7b70ab752ec9d19e205bf8c9e88dc4e40a6" exitCode=2
Sep 16 19:35:31.144289 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:31.144118    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" event={"ID":"d2a81649-573f-4427-891d-050cd947ac64","Type":"ContainerDied","Data":"ce2a0655b77c7223c22a8318069cd7b70ab752ec9d19e205bf8c9e88dc4e40a6"}
Sep 16 19:35:31.742022 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:31.741978    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Sep 16 19:35:34.811174 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:34.811132    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 16 19:35:35.164660 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.164625    2584 generic.go:356] "Generic (PLEG): container finished" podID="d2a81649-573f-4427-891d-050cd947ac64" containerID="51245a9f291a8cd1bc42ad33878bafd69feb0edbcfed944de9f0de7cb55ea97d" exitCode=0
Sep 16 19:35:35.164801 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.164696    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" event={"ID":"d2a81649-573f-4427-891d-050cd947ac64","Type":"ContainerDied","Data":"51245a9f291a8cd1bc42ad33878bafd69feb0edbcfed944de9f0de7cb55ea97d"}
Sep 16 19:35:35.187489 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.187469    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:35:35.262949 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.262920    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-kube-rbac-proxy-sar-config\") pod \"d2a81649-573f-4427-891d-050cd947ac64\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") "
Sep 16 19:35:35.263107 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.262962    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-cabundle-cert\") pod \"d2a81649-573f-4427-891d-050cd947ac64\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") "
Sep 16 19:35:35.263107 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.262984    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls\") pod \"d2a81649-573f-4427-891d-050cd947ac64\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") "
Sep 16 19:35:35.263107 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.263020    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a81649-573f-4427-891d-050cd947ac64-kserve-provision-location\") pod \"d2a81649-573f-4427-891d-050cd947ac64\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") "
Sep 16 19:35:35.263107 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.263065    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmzbm\" (UniqueName: \"kubernetes.io/projected/d2a81649-573f-4427-891d-050cd947ac64-kube-api-access-wmzbm\") pod \"d2a81649-573f-4427-891d-050cd947ac64\" (UID: \"d2a81649-573f-4427-891d-050cd947ac64\") "
Sep 16 19:35:35.263419 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.263386    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a81649-573f-4427-891d-050cd947ac64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2a81649-573f-4427-891d-050cd947ac64" (UID: "d2a81649-573f-4427-891d-050cd947ac64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:35.263419 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.263406    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2a81649-573f-4427-891d-050cd947ac64" (UID: "d2a81649-573f-4427-891d-050cd947ac64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:35.263585 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.263507    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2a81649-573f-4427-891d-050cd947ac64" (UID: "d2a81649-573f-4427-891d-050cd947ac64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:35.265125 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.265096    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a81649-573f-4427-891d-050cd947ac64-kube-api-access-wmzbm" (OuterVolumeSpecName: "kube-api-access-wmzbm") pod "d2a81649-573f-4427-891d-050cd947ac64" (UID: "d2a81649-573f-4427-891d-050cd947ac64"). InnerVolumeSpecName "kube-api-access-wmzbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:35.265125 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.265117    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2a81649-573f-4427-891d-050cd947ac64" (UID: "d2a81649-573f-4427-891d-050cd947ac64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:35.364226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.364163    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:35:35.364226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.364193    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a81649-573f-4427-891d-050cd947ac64-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:35:35.364226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.364205    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a81649-573f-4427-891d-050cd947ac64-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:35:35.364226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.364212    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a81649-573f-4427-891d-050cd947ac64-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:35:35.364226 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:35.364221    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmzbm\" (UniqueName: \"kubernetes.io/projected/d2a81649-573f-4427-891d-050cd947ac64-kube-api-access-wmzbm\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:35:36.171842 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:36.171810    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p" event={"ID":"d2a81649-573f-4427-891d-050cd947ac64","Type":"ContainerDied","Data":"571d0bc9995c0b4d8ebc9b0d7e0ec899f5bf8cb3443f027a3180b47ac8fc4aa8"}
Sep 16 19:35:36.171842 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:36.171841    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"
Sep 16 19:35:36.172372 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:36.171890    2584 scope.go:117] "RemoveContainer" containerID="ce2a0655b77c7223c22a8318069cd7b70ab752ec9d19e205bf8c9e88dc4e40a6"
Sep 16 19:35:36.180382 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:36.180354    2584 scope.go:117] "RemoveContainer" containerID="51245a9f291a8cd1bc42ad33878bafd69feb0edbcfed944de9f0de7cb55ea97d"
Sep 16 19:35:36.187628 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:36.187612    2584 scope.go:117] "RemoveContainer" containerID="fc525f4856b575f1daa7d7be301a176f6b0b0df96d91bd182d36ec708c01d798"
Sep 16 19:35:36.192431 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:36.192411    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"]
Sep 16 19:35:36.199602 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:36.199582    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74b76c8bb4-tdg7p"]
Sep 16 19:35:37.013430 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:37.013394    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:35:37.816281 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:37.816242    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a81649-573f-4427-891d-050cd947ac64" path="/var/lib/kubelet/pods/d2a81649-573f-4427-891d-050cd947ac64/volumes"
Sep 16 19:35:47.013902 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:47.013864    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:35:57.014194 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:35:57.014151    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:36:07.013724 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:07.013681    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:36:13.811561 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:13.811517    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 16 19:36:23.815904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:23.815873    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:36:28.125803 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:28.125762    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"]
Sep 16 19:36:28.126293 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:28.125981    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container" containerID="cri-o://6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac" gracePeriod=30
Sep 16 19:36:28.126293 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:28.126067    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kube-rbac-proxy" containerID="cri-o://ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38" gracePeriod=30
Sep 16 19:36:28.405286 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:28.405171    2584 generic.go:356] "Generic (PLEG): container finished" podID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerID="ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38" exitCode=2
Sep 16 19:36:28.405286 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:28.405232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" event={"ID":"26e79c9c-1f5b-4206-966b-f15b02d8ebfc","Type":"ContainerDied","Data":"ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38"}
Sep 16 19:36:31.583370 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.583344    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:36:31.725726 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.725628    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-proxy-tls\") pod \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") "
Sep 16 19:36:31.725726 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.725683    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-cabundle-cert\") pod \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") "
Sep 16 19:36:31.725726 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.725711    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhrrm\" (UniqueName: \"kubernetes.io/projected/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-api-access-zhrrm\") pod \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") "
Sep 16 19:36:31.725726 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.725731    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-rbac-proxy-sar-config\") pod \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") "
Sep 16 19:36:31.726079 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.725778    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kserve-provision-location\") pod \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\" (UID: \"26e79c9c-1f5b-4206-966b-f15b02d8ebfc\") "
Sep 16 19:36:31.726161 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.726133    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26e79c9c-1f5b-4206-966b-f15b02d8ebfc" (UID: "26e79c9c-1f5b-4206-966b-f15b02d8ebfc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:31.726161 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.726147    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26e79c9c-1f5b-4206-966b-f15b02d8ebfc" (UID: "26e79c9c-1f5b-4206-966b-f15b02d8ebfc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:31.726250 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.726140    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26e79c9c-1f5b-4206-966b-f15b02d8ebfc" (UID: "26e79c9c-1f5b-4206-966b-f15b02d8ebfc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:31.727866 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.727847    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26e79c9c-1f5b-4206-966b-f15b02d8ebfc" (UID: "26e79c9c-1f5b-4206-966b-f15b02d8ebfc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:31.727921 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.727900    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-api-access-zhrrm" (OuterVolumeSpecName: "kube-api-access-zhrrm") pod "26e79c9c-1f5b-4206-966b-f15b02d8ebfc" (UID: "26e79c9c-1f5b-4206-966b-f15b02d8ebfc"). InnerVolumeSpecName "kube-api-access-zhrrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:31.826935 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.826913    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:36:31.826935 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.826936    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:36:31.827083 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.826949    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:36:31.827083 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.826961    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhrrm\" (UniqueName: \"kubernetes.io/projected/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-api-access-zhrrm\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:36:31.827083 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:31.826972    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26e79c9c-1f5b-4206-966b-f15b02d8ebfc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:36:32.425796 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.425753    2584 generic.go:356] "Generic (PLEG): container finished" podID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerID="6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac" exitCode=0
Sep 16 19:36:32.426004 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.425823    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"
Sep 16 19:36:32.426004 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.425843    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" event={"ID":"26e79c9c-1f5b-4206-966b-f15b02d8ebfc","Type":"ContainerDied","Data":"6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac"}
Sep 16 19:36:32.426004 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.425897    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6" event={"ID":"26e79c9c-1f5b-4206-966b-f15b02d8ebfc","Type":"ContainerDied","Data":"3c6ea63c0494a33248edafedbeacaf22181cf241b1055a4a149aed57ab60fdff"}
Sep 16 19:36:32.426004 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.425923    2584 scope.go:117] "RemoveContainer" containerID="ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38"
Sep 16 19:36:32.434199 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.434180    2584 scope.go:117] "RemoveContainer" containerID="6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac"
Sep 16 19:36:32.441642 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.441618    2584 scope.go:117] "RemoveContainer" containerID="91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663"
Sep 16 19:36:32.445167 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.445143    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"]
Sep 16 19:36:32.449456 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.449434    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9dbh6"]
Sep 16 19:36:32.450527 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.450508    2584 scope.go:117] "RemoveContainer" containerID="ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38"
Sep 16 19:36:32.450833 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:36:32.450810    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38\": container with ID starting with ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38 not found: ID does not exist" containerID="ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38"
Sep 16 19:36:32.450909 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.450846    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38"} err="failed to get container status \"ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38\": rpc error: code = NotFound desc = could not find container \"ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38\": container with ID starting with ab0632029e122fb8a05eb97615398efc50b5ef5357752620f66b41a4f192ac38 not found: ID does not exist"
Sep 16 19:36:32.450909 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.450870    2584 scope.go:117] "RemoveContainer" containerID="6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac"
Sep 16 19:36:32.451131 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:36:32.451112    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac\": container with ID starting with 6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac not found: ID does not exist" containerID="6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac"
Sep 16 19:36:32.451191 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.451140    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac"} err="failed to get container status \"6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac\": rpc error: code = NotFound desc = could not find container \"6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac\": container with ID starting with 6a1949c5724609c8b82b2ab27dfac168eacbdf379523498b9281ea16806d9cac not found: ID does not exist"
Sep 16 19:36:32.451191 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.451159    2584 scope.go:117] "RemoveContainer" containerID="91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663"
Sep 16 19:36:32.451436 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:36:32.451419    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663\": container with ID starting with 91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663 not found: ID does not exist" containerID="91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663"
Sep 16 19:36:32.451498 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:32.451440    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663"} err="failed to get container status \"91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663\": rpc error: code = NotFound desc = could not find container \"91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663\": container with ID starting with 91da7681a1c98a0d37c808eb7253aced8fca0ecc5252150ef5f831556c76b663 not found: ID does not exist"
Sep 16 19:36:33.330460 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330431    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"]
Sep 16 19:36:33.330808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330761    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kube-rbac-proxy"
Sep 16 19:36:33.330808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330773    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kube-rbac-proxy"
Sep 16 19:36:33.330808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330787    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="storage-initializer"
Sep 16 19:36:33.330808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330793    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="storage-initializer"
Sep 16 19:36:33.330808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330806    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="storage-initializer"
Sep 16 19:36:33.330808 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330811    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="storage-initializer"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330818    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330823    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330830    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330835    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330840    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kube-rbac-proxy"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330846    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kube-rbac-proxy"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330891    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kserve-container"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330900    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kserve-container"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330908    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" containerName="kube-rbac-proxy"
Sep 16 19:36:33.331002 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.330914    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a81649-573f-4427-891d-050cd947ac64" containerName="kube-rbac-proxy"
Sep 16 19:36:33.335506 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.335485    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.339746 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.339720    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:36:33.340905 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.340884    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:36:33.341034 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.340915    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:36:33.341034 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.340932    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:36:33.341197 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.340940    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:36:33.341289 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.340891    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 16 19:36:33.342313 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.342294    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:36:33.351157 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.351135    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"]
Sep 16 19:36:33.439100 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.439072    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.439296 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.439119    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.439296 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.439152    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.439296 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.439187    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.439296 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.439208    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9lv\" (UniqueName: \"kubernetes.io/projected/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-api-access-2l9lv\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.540223 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.540197    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.540409 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.540241    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.540409 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.540301    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.540409 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.540322    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9lv\" (UniqueName: \"kubernetes.io/projected/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-api-access-2l9lv\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.540409 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.540349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.540760 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.540737    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.540919 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.540891    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.541061 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.541038    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.543057 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.543036    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.557896 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.557873    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9lv\" (UniqueName: \"kubernetes.io/projected/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-api-access-2l9lv\") pod \"isvc-pmml-runtime-predictor-885fc8777-zzk2p\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.646602 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.646520    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:33.816762 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.816722    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e79c9c-1f5b-4206-966b-f15b02d8ebfc" path="/var/lib/kubelet/pods/26e79c9c-1f5b-4206-966b-f15b02d8ebfc/volumes"
Sep 16 19:36:33.917498 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:33.917470    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"]
Sep 16 19:36:33.919508 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:36:33.919482    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31799d9b_05e3_488d_b49b_9b4fcc93bbbd.slice/crio-4639b194cc9378cdff25529ac326fcd5db53bdd79f47f4ba1bcfdb4f5c615523 WatchSource:0}: Error finding container 4639b194cc9378cdff25529ac326fcd5db53bdd79f47f4ba1bcfdb4f5c615523: Status 404 returned error can't find the container with id 4639b194cc9378cdff25529ac326fcd5db53bdd79f47f4ba1bcfdb4f5c615523
Sep 16 19:36:34.437606 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:34.437575    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" event={"ID":"31799d9b-05e3-488d-b49b-9b4fcc93bbbd","Type":"ContainerStarted","Data":"0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419"}
Sep 16 19:36:34.438083 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:34.437660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" event={"ID":"31799d9b-05e3-488d-b49b-9b4fcc93bbbd","Type":"ContainerStarted","Data":"4639b194cc9378cdff25529ac326fcd5db53bdd79f47f4ba1bcfdb4f5c615523"}
Sep 16 19:36:38.457224 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:38.457190    2584 generic.go:356] "Generic (PLEG): container finished" podID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerID="0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419" exitCode=0
Sep 16 19:36:38.457687 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:38.457228    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" event={"ID":"31799d9b-05e3-488d-b49b-9b4fcc93bbbd","Type":"ContainerDied","Data":"0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419"}
Sep 16 19:36:39.464180 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:39.464147    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" event={"ID":"31799d9b-05e3-488d-b49b-9b4fcc93bbbd","Type":"ContainerStarted","Data":"f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9"}
Sep 16 19:36:39.464578 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:39.464206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" event={"ID":"31799d9b-05e3-488d-b49b-9b4fcc93bbbd","Type":"ContainerStarted","Data":"db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585"}
Sep 16 19:36:39.464578 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:39.464456    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:39.486178 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:39.486128    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podStartSLOduration=2.503214489 podStartE2EDuration="6.486112591s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.982898102s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:39.484787252 +0000 UTC m=+3130.151726409" watchObservedRunningTime="2026-09-16 19:36:39.486112591 +0000 UTC m=+3130.153051736"
Sep 16 19:36:40.473756 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:40.473722    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:40.475118 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:40.475088    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:36:41.478484 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:41.478442    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:36:46.483098 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:46.483071    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:36:46.483551 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:46.483526    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:36:56.483484 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:36:56.483446    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:37:06.483972 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:37:06.483928    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:37:16.483496 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:37:16.483447    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:37:26.484246 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:37:26.484205    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:37:36.483972 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:37:36.483931    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:37:46.483632 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:37:46.483591    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:37:56.484948 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:37:56.484919    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:38:04.443373 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:04.443337    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"]
Sep 16 19:38:04.443827 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:04.443659    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" containerID="cri-o://db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585" gracePeriod=30
Sep 16 19:38:04.443827 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:04.443755    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kube-rbac-proxy" containerID="cri-o://f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9" gracePeriod=30
Sep 16 19:38:04.856034 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:04.855952    2584 generic.go:356] "Generic (PLEG): container finished" podID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerID="f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9" exitCode=2
Sep 16 19:38:04.856170 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:04.856028    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" event={"ID":"31799d9b-05e3-488d-b49b-9b4fcc93bbbd","Type":"ContainerDied","Data":"f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9"}
Sep 16 19:38:06.478710 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:06.478673    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Sep 16 19:38:06.483917 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:06.483887    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 16 19:38:07.483962 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.483941    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:38:07.574880 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.574815    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-rbac-proxy-sar-config\") pod \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") "
Sep 16 19:38:07.574880 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.574858    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kserve-provision-location\") pod \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") "
Sep 16 19:38:07.575093 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.574892    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-proxy-tls\") pod \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") "
Sep 16 19:38:07.575093 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.574923    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-cabundle-cert\") pod \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") "
Sep 16 19:38:07.575093 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.574945    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l9lv\" (UniqueName: \"kubernetes.io/projected/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-api-access-2l9lv\") pod \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\" (UID: \"31799d9b-05e3-488d-b49b-9b4fcc93bbbd\") "
Sep 16 19:38:07.575258 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.575143    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31799d9b-05e3-488d-b49b-9b4fcc93bbbd" (UID: "31799d9b-05e3-488d-b49b-9b4fcc93bbbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:07.575348 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.575294    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31799d9b-05e3-488d-b49b-9b4fcc93bbbd" (UID: "31799d9b-05e3-488d-b49b-9b4fcc93bbbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:38:07.575348 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.575316    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31799d9b-05e3-488d-b49b-9b4fcc93bbbd" (UID: "31799d9b-05e3-488d-b49b-9b4fcc93bbbd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:38:07.576965 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.576945    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31799d9b-05e3-488d-b49b-9b4fcc93bbbd" (UID: "31799d9b-05e3-488d-b49b-9b4fcc93bbbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:07.577029 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.576961    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-api-access-2l9lv" (OuterVolumeSpecName: "kube-api-access-2l9lv") pod "31799d9b-05e3-488d-b49b-9b4fcc93bbbd" (UID: "31799d9b-05e3-488d-b49b-9b4fcc93bbbd"). InnerVolumeSpecName "kube-api-access-2l9lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:07.675774 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.675749    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:38:07.675774 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.675772    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l9lv\" (UniqueName: \"kubernetes.io/projected/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-api-access-2l9lv\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:38:07.675904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.675784    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:38:07.675904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.675793    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:38:07.675904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.675802    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31799d9b-05e3-488d-b49b-9b4fcc93bbbd-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:38:07.868707 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.868641    2584 generic.go:356] "Generic (PLEG): container finished" podID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerID="db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585" exitCode=0
Sep 16 19:38:07.868707 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.868701    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"
Sep 16 19:38:07.868892 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.868717    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" event={"ID":"31799d9b-05e3-488d-b49b-9b4fcc93bbbd","Type":"ContainerDied","Data":"db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585"}
Sep 16 19:38:07.868892 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.868817    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p" event={"ID":"31799d9b-05e3-488d-b49b-9b4fcc93bbbd","Type":"ContainerDied","Data":"4639b194cc9378cdff25529ac326fcd5db53bdd79f47f4ba1bcfdb4f5c615523"}
Sep 16 19:38:07.868892 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.868855    2584 scope.go:117] "RemoveContainer" containerID="f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9"
Sep 16 19:38:07.877120 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.877104    2584 scope.go:117] "RemoveContainer" containerID="db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585"
Sep 16 19:38:07.884554 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.884537    2584 scope.go:117] "RemoveContainer" containerID="0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419"
Sep 16 19:38:07.885578 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.885555    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"]
Sep 16 19:38:07.891496 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.891475    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-zzk2p"]
Sep 16 19:38:07.892660 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.892647    2584 scope.go:117] "RemoveContainer" containerID="f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9"
Sep 16 19:38:07.892918 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:38:07.892900    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9\": container with ID starting with f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9 not found: ID does not exist" containerID="f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9"
Sep 16 19:38:07.892967 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.892929    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9"} err="failed to get container status \"f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9\": rpc error: code = NotFound desc = could not find container \"f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9\": container with ID starting with f0768588e6f89c147eeef97aa7040d6be07157f40a3128fe6f1a8ac5051093e9 not found: ID does not exist"
Sep 16 19:38:07.892967 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.892947    2584 scope.go:117] "RemoveContainer" containerID="db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585"
Sep 16 19:38:07.893175 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:38:07.893156    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585\": container with ID starting with db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585 not found: ID does not exist" containerID="db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585"
Sep 16 19:38:07.893240 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.893184    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585"} err="failed to get container status \"db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585\": rpc error: code = NotFound desc = could not find container \"db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585\": container with ID starting with db898f2f759e73fac976ef1b75c168276dc37c303f8e6f76d2b8a3ef99d74585 not found: ID does not exist"
Sep 16 19:38:07.893240 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.893206    2584 scope.go:117] "RemoveContainer" containerID="0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419"
Sep 16 19:38:07.893451 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:38:07.893437    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419\": container with ID starting with 0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419 not found: ID does not exist" containerID="0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419"
Sep 16 19:38:07.893499 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:07.893454    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419"} err="failed to get container status \"0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419\": rpc error: code = NotFound desc = could not find container \"0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419\": container with ID starting with 0ba396373beedf618c2dbd368f9373d7ed61b920939bdf9f1d1845bbe03e7419 not found: ID does not exist"
Sep 16 19:38:09.638356 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.638321    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"]
Sep 16 19:38:09.638837 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.638820    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="storage-initializer"
Sep 16 19:38:09.638894 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.638840    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="storage-initializer"
Sep 16 19:38:09.638894 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.638852    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container"
Sep 16 19:38:09.638894 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.638861    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container"
Sep 16 19:38:09.638894 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.638870    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kube-rbac-proxy"
Sep 16 19:38:09.638894 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.638878    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kube-rbac-proxy"
Sep 16 19:38:09.639069 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.638950    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kserve-container"
Sep 16 19:38:09.639069 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.638964    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" containerName="kube-rbac-proxy"
Sep 16 19:38:09.643904 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.643883    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.646692 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.646656    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-4c492e-kube-rbac-proxy-sar-config\""
Sep 16 19:38:09.646692 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.646685    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:38:09.646882 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.646776    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:38:09.646882 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.646793    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:38:09.646996 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.646898    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-4c492e-predictor-serving-cert\""
Sep 16 19:38:09.647708 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.647693    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:38:09.647756 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.647702    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:38:09.651573 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.651550    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"]
Sep 16 19:38:09.689572 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.689549    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-proxy-tls\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.689681 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.689579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kserve-provision-location\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.689681 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.689616    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-rbac-proxy-sar-config\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.689820 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.689684    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dptbp\" (UniqueName: \"kubernetes.io/projected/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-api-access-dptbp\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.689820 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.689734    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-cabundle-cert\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.790025 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.790001    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-rbac-proxy-sar-config\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.790149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.790038    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dptbp\" (UniqueName: \"kubernetes.io/projected/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-api-access-dptbp\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.790149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.790070    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-cabundle-cert\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.790149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.790104    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-proxy-tls\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.790149 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.790121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kserve-provision-location\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.790404 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:38:09.790235    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-4c492e-predictor-serving-cert: secret "isvc-primary-4c492e-predictor-serving-cert" not found
Sep 16 19:38:09.790404 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:38:09.790313    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-proxy-tls podName:7b72de6c-6ea8-4fb8-9dac-d74f425c2868 nodeName:}" failed. No retries permitted until 2026-09-16 19:38:10.290294099 +0000 UTC m=+3220.957233224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-proxy-tls") pod "isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" (UID: "7b72de6c-6ea8-4fb8-9dac-d74f425c2868") : secret "isvc-primary-4c492e-predictor-serving-cert" not found
Sep 16 19:38:09.790589 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.790565    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kserve-provision-location\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.790719 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.790703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-rbac-proxy-sar-config\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.790775 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.790762    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-cabundle-cert\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.798797 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.798775    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dptbp\" (UniqueName: \"kubernetes.io/projected/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-api-access-dptbp\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:09.816425 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:09.816401    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31799d9b-05e3-488d-b49b-9b4fcc93bbbd" path="/var/lib/kubelet/pods/31799d9b-05e3-488d-b49b-9b4fcc93bbbd/volumes"
Sep 16 19:38:10.293532 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:10.293493    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-proxy-tls\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:10.295880 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:10.295846    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-proxy-tls\") pod \"isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") " pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:10.556346 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:10.556230    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:10.830310 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:10.830220    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"]
Sep 16 19:38:10.833220 ip-10-0-139-120 kubenswrapper[2584]: W0916 19:38:10.833190    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b72de6c_6ea8_4fb8_9dac_d74f425c2868.slice/crio-7fdb560186da5b35e9dddf26bef43f22cd4d09e8bb8e5346399dab8287102594 WatchSource:0}: Error finding container 7fdb560186da5b35e9dddf26bef43f22cd4d09e8bb8e5346399dab8287102594: Status 404 returned error can't find the container with id 7fdb560186da5b35e9dddf26bef43f22cd4d09e8bb8e5346399dab8287102594
Sep 16 19:38:10.886522 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:10.886497    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" event={"ID":"7b72de6c-6ea8-4fb8-9dac-d74f425c2868","Type":"ContainerStarted","Data":"7fdb560186da5b35e9dddf26bef43f22cd4d09e8bb8e5346399dab8287102594"}
Sep 16 19:38:11.892589 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:11.892551    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" event={"ID":"7b72de6c-6ea8-4fb8-9dac-d74f425c2868","Type":"ContainerStarted","Data":"e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a"}
Sep 16 19:38:14.906246 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:14.906167    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerID="e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a" exitCode=0
Sep 16 19:38:14.906604 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:14.906240    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" event={"ID":"7b72de6c-6ea8-4fb8-9dac-d74f425c2868","Type":"ContainerDied","Data":"e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a"}
Sep 16 19:38:15.912773 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:15.912736    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" event={"ID":"7b72de6c-6ea8-4fb8-9dac-d74f425c2868","Type":"ContainerStarted","Data":"ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a"}
Sep 16 19:38:15.913194 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:15.912812    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" event={"ID":"7b72de6c-6ea8-4fb8-9dac-d74f425c2868","Type":"ContainerStarted","Data":"f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb"}
Sep 16 19:38:15.913194 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:15.913044    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:15.933503 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:15.933460    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podStartSLOduration=3.249889309 podStartE2EDuration="6.933444958s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.683555649s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:15.93246348 +0000 UTC m=+3226.599402643" watchObservedRunningTime="2026-09-16 19:38:15.933444958 +0000 UTC m=+3226.600384104"
Sep 16 19:38:16.917131 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:16.917100    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:16.918378 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:16.918351    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:38:17.920811 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:17.920773    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:38:22.925291 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:22.925203    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:38:22.925673 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:22.925647    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:38:32.925806 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:32.925753    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:38:42.925882 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:42.925837    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:38:52.926540 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:38:52.926496    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:39:02.925840 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:39:02.925803    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:39:12.926461 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:39:12.926420    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:39:22.926210 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:39:22.926167    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:39:32.927015 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:39:32.926972    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:40:10.911890 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:10.911852    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"]
Sep 16 19:40:10.912440 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:10.912118    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" containerID="cri-o://f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb" gracePeriod=30
Sep 16 19:40:10.912440 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:10.912159    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kube-rbac-proxy" containerID="cri-o://ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a" gracePeriod=30
Sep 16 19:40:11.424047 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:11.424017    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerID="ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a" exitCode=2
Sep 16 19:40:11.424215 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:11.424056    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" event={"ID":"7b72de6c-6ea8-4fb8-9dac-d74f425c2868","Type":"ContainerDied","Data":"ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a"}
Sep 16 19:40:12.921155 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:12.921115    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Sep 16 19:40:12.926501 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:12.926475    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 16 19:40:15.070861 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.070838    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:40:15.232741 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.232717    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-rbac-proxy-sar-config\") pod \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") "
Sep 16 19:40:15.232891 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.232752    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kserve-provision-location\") pod \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") "
Sep 16 19:40:15.232891 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.232796    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-proxy-tls\") pod \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") "
Sep 16 19:40:15.232994 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.232891    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dptbp\" (UniqueName: \"kubernetes.io/projected/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-api-access-dptbp\") pod \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") "
Sep 16 19:40:15.232994 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.232936    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-cabundle-cert\") pod \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\" (UID: \"7b72de6c-6ea8-4fb8-9dac-d74f425c2868\") "
Sep 16 19:40:15.233130 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.233101    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b72de6c-6ea8-4fb8-9dac-d74f425c2868" (UID: "7b72de6c-6ea8-4fb8-9dac-d74f425c2868"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:40:15.233195 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.233121    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b72de6c-6ea8-4fb8-9dac-d74f425c2868" (UID: "7b72de6c-6ea8-4fb8-9dac-d74f425c2868"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:15.233299 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.233279    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:40:15.233386 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.233304    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kserve-provision-location\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:40:15.233386 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.233365    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b72de6c-6ea8-4fb8-9dac-d74f425c2868" (UID: "7b72de6c-6ea8-4fb8-9dac-d74f425c2868"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:40:15.234858 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.234831    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b72de6c-6ea8-4fb8-9dac-d74f425c2868" (UID: "7b72de6c-6ea8-4fb8-9dac-d74f425c2868"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:40:15.234952 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.234894    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-api-access-dptbp" (OuterVolumeSpecName: "kube-api-access-dptbp") pod "7b72de6c-6ea8-4fb8-9dac-d74f425c2868" (UID: "7b72de6c-6ea8-4fb8-9dac-d74f425c2868"). InnerVolumeSpecName "kube-api-access-dptbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:40:15.333835 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.333803    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-proxy-tls\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:40:15.333835 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.333832    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dptbp\" (UniqueName: \"kubernetes.io/projected/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-kube-api-access-dptbp\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:40:15.333835 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.333843    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b72de6c-6ea8-4fb8-9dac-d74f425c2868-cabundle-cert\") on node \"ip-10-0-139-120.ec2.internal\" DevicePath \"\""
Sep 16 19:40:15.450596 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.450562    2584 generic.go:356] "Generic (PLEG): container finished" podID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerID="f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb" exitCode=0
Sep 16 19:40:15.450753 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.450599    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" event={"ID":"7b72de6c-6ea8-4fb8-9dac-d74f425c2868","Type":"ContainerDied","Data":"f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb"}
Sep 16 19:40:15.450753 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.450622    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"
Sep 16 19:40:15.450753 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.450665    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj" event={"ID":"7b72de6c-6ea8-4fb8-9dac-d74f425c2868","Type":"ContainerDied","Data":"7fdb560186da5b35e9dddf26bef43f22cd4d09e8bb8e5346399dab8287102594"}
Sep 16 19:40:15.450753 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.450691    2584 scope.go:117] "RemoveContainer" containerID="ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a"
Sep 16 19:40:15.459947 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.459930    2584 scope.go:117] "RemoveContainer" containerID="f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb"
Sep 16 19:40:15.466978 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.466962    2584 scope.go:117] "RemoveContainer" containerID="e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a"
Sep 16 19:40:15.475027 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.475004    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"]
Sep 16 19:40:15.475619 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.475601    2584 scope.go:117] "RemoveContainer" containerID="ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a"
Sep 16 19:40:15.476057 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:40:15.476010    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a\": container with ID starting with ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a not found: ID does not exist" containerID="ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a"
Sep 16 19:40:15.476216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.476068    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a"} err="failed to get container status \"ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a\": rpc error: code = NotFound desc = could not find container \"ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a\": container with ID starting with ac5e69a9e4d9dba0deb3101789f6152c04277a7b917c4d7cf8978d0fe668476a not found: ID does not exist"
Sep 16 19:40:15.476216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.476092    2584 scope.go:117] "RemoveContainer" containerID="f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb"
Sep 16 19:40:15.476514 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:40:15.476485    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb\": container with ID starting with f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb not found: ID does not exist" containerID="f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb"
Sep 16 19:40:15.476621 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.476516    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb"} err="failed to get container status \"f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb\": rpc error: code = NotFound desc = could not find container \"f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb\": container with ID starting with f6264f04185b736ed69542c6d3834ef46afb0e6cad92ed341750d7fb54020ddb not found: ID does not exist"
Sep 16 19:40:15.476621 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.476536    2584 scope.go:117] "RemoveContainer" containerID="e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a"
Sep 16 19:40:15.476949 ip-10-0-139-120 kubenswrapper[2584]: E0916 19:40:15.476903    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a\": container with ID starting with e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a not found: ID does not exist" containerID="e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a"
Sep 16 19:40:15.477019 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.476945    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a"} err="failed to get container status \"e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a\": rpc error: code = NotFound desc = could not find container \"e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a\": container with ID starting with e950daf565baf0780c749d0055fea74c9a64cf7abb52f7caee4028b1a2462a9a not found: ID does not exist"
Sep 16 19:40:15.477960 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.477944    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-4c492e-predictor-677d9dbd4c-cvrhj"]
Sep 16 19:40:15.816038 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:15.816010    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" path="/var/lib/kubelet/pods/7b72de6c-6ea8-4fb8-9dac-d74f425c2868/volumes"
Sep 16 19:40:56.964838 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:56.964809    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6cn4c_91214ab9-3932-439b-821e-5746a112e114/global-pull-secret-syncer/0.log"
Sep 16 19:40:57.141652 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:57.141618    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w9rjm_d5d14846-8078-4eed-972f-75e7fff7d7cb/konnectivity-agent/0.log"
Sep 16 19:40:57.213373 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:40:57.213343    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-120.ec2.internal_8611bbd0ccde75189058c67a0a76ec36/haproxy/0.log"
Sep 16 19:41:01.759298 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:01.759252    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cbmd_b04c2df7-04c9-416f-9da1-99d27b8d9142/node-exporter/0.log"
Sep 16 19:41:01.788874 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:01.788852    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cbmd_b04c2df7-04c9-416f-9da1-99d27b8d9142/kube-rbac-proxy/0.log"
Sep 16 19:41:01.826616 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:01.826596    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cbmd_b04c2df7-04c9-416f-9da1-99d27b8d9142/init-textfile/0.log"
Sep 16 19:41:01.953849 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:01.953825    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lj8dn_f5862522-1ff7-47d1-b29d-24a445296331/kube-rbac-proxy-main/0.log"
Sep 16 19:41:01.979534 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:01.979511    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lj8dn_f5862522-1ff7-47d1-b29d-24a445296331/kube-rbac-proxy-self/0.log"
Sep 16 19:41:02.007082 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:02.007053    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-lj8dn_f5862522-1ff7-47d1-b29d-24a445296331/openshift-state-metrics/0.log"
Sep 16 19:41:02.246847 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:02.246829    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gn8d8_7d4bf852-bbc0-4429-a0e6-c9a723915f11/prometheus-operator/0.log"
Sep 16 19:41:02.273780 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:02.273749    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gn8d8_7d4bf852-bbc0-4429-a0e6-c9a723915f11/kube-rbac-proxy/0.log"
Sep 16 19:41:03.979875 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:03.979843    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-2275q_e13f7d12-6a82-469f-8c78-52ef0e7af2b3/networking-console-plugin/0.log"
Sep 16 19:41:04.998755 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:04.998723    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-865db6676c-zj9th_19f2ccc3-f370-41f8-a248-ae3d60243d64/console/0.log"
Sep 16 19:41:05.033859 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.033835    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-lf2z7_c34fffd4-724f-41cd-8277-f53e80ec0b57/download-server/0.log"
Sep 16 19:41:05.608528 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.608500    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-n26xz_16d42993-1963-4a2b-851b-cd2bd9dfb477/volume-data-source-validator/0.log"
Sep 16 19:41:05.878601 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.878522    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"]
Sep 16 19:41:05.878855 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.878843    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container"
Sep 16 19:41:05.878898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.878856    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container"
Sep 16 19:41:05.878898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.878869    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kube-rbac-proxy"
Sep 16 19:41:05.878898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.878874    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kube-rbac-proxy"
Sep 16 19:41:05.878898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.878888    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="storage-initializer"
Sep 16 19:41:05.878898 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.878894    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="storage-initializer"
Sep 16 19:41:05.879057 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.878947    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kserve-container"
Sep 16 19:41:05.879057 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.878955    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b72de6c-6ea8-4fb8-9dac-d74f425c2868" containerName="kube-rbac-proxy"
Sep 16 19:41:05.883108 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.883090    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:05.885552 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.885524    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t4xfr\"/\"default-dockercfg-6fc9k\""
Sep 16 19:41:05.885672 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.885551    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4xfr\"/\"openshift-service-ca.crt\""
Sep 16 19:41:05.886576 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.886558    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4xfr\"/\"kube-root-ca.crt\""
Sep 16 19:41:05.892883 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:05.892857    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"]
Sep 16 19:41:06.020641 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.020604    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-proc\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.021005 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.020650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swz69\" (UniqueName: \"kubernetes.io/projected/bd1566ba-abdc-449d-ac7b-60b7080a6c68-kube-api-access-swz69\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.021005 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.020676    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-sys\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.021005 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.020752    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-lib-modules\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.021005 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.020845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-podres\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.122036 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.122001    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swz69\" (UniqueName: \"kubernetes.io/projected/bd1566ba-abdc-449d-ac7b-60b7080a6c68-kube-api-access-swz69\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.122036 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.122041    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-sys\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.122216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.122062    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-lib-modules\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.122216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.122112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-podres\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.122216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.122135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-proc\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.122216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.122153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-sys\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.122216 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.122196    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-proc\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.122405 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.122246    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-podres\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.122405 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.122250    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd1566ba-abdc-449d-ac7b-60b7080a6c68-lib-modules\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.130482 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.130427    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swz69\" (UniqueName: \"kubernetes.io/projected/bd1566ba-abdc-449d-ac7b-60b7080a6c68-kube-api-access-swz69\") pod \"perf-node-gather-daemonset-zvmlb\" (UID: \"bd1566ba-abdc-449d-ac7b-60b7080a6c68\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.193564 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.193539    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.493424 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.493399    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"]
Sep 16 19:41:06.639214 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.639191    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w9zrj_5d19db54-ba2c-4ea5-8134-a08ea264e3a7/dns/0.log"
Sep 16 19:41:06.659811 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.659792    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w9zrj_5d19db54-ba2c-4ea5-8134-a08ea264e3a7/kube-rbac-proxy/0.log"
Sep 16 19:41:06.683227 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.683197    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb" event={"ID":"bd1566ba-abdc-449d-ac7b-60b7080a6c68","Type":"ContainerStarted","Data":"f41024da3de085b0125d0cf6d8fc2e43882dd122021bdebec300ff1bc0d3ea08"}
Sep 16 19:41:06.683349 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.683252    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb" event={"ID":"bd1566ba-abdc-449d-ac7b-60b7080a6c68","Type":"ContainerStarted","Data":"2bcf4eb7a0c5f13b3fdf48ea3449ea80404b32960033629d03f218e4f1d39e2b"}
Sep 16 19:41:06.683349 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.683302    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:06.700349 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.700312    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb" podStartSLOduration=1.700296392 podStartE2EDuration="1.700296392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:06.69862776 +0000 UTC m=+3397.365566925" watchObservedRunningTime="2026-09-16 19:41:06.700296392 +0000 UTC m=+3397.367235590"
Sep 16 19:41:06.707299 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:06.707278    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hh97b_77743e7d-141f-40d6-9fa8-c58c087a0a6a/dns-node-resolver/0.log"
Sep 16 19:41:07.215596 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:07.215566    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-549f4b54cd-lvknp_ba382a88-6fe6-42f0-a236-ed3bf71e4078/registry/0.log"
Sep 16 19:41:07.237554 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:07.237526    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dlqjl_dc2d6470-4c6a-4a11-9387-5519d9858bc2/node-ca/0.log"
Sep 16 19:41:08.076799 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:08.076752    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-646544f686-74gxd_4d092af1-0b63-4f2c-89f2-6cffbe76bdbd/router/0.log"
Sep 16 19:41:08.477150 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:08.477127    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rvnq9_ed420e76-3215-428e-aa07-27c0047a9619/serve-healthcheck-canary/0.log"
Sep 16 19:41:08.838010 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:08.837939    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-h97rl_3674164a-4141-4d00-ba6a-45fff63302fb/insights-operator/0.log"
Sep 16 19:41:08.838983 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:08.838967    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-h97rl_3674164a-4141-4d00-ba6a-45fff63302fb/insights-operator/1.log"
Sep 16 19:41:09.004501 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:09.004477    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdz2g_b56eaab7-f00c-48cf-841d-d46f859324e8/kube-rbac-proxy/0.log"
Sep 16 19:41:09.023612 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:09.023582    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdz2g_b56eaab7-f00c-48cf-841d-d46f859324e8/exporter/0.log"
Sep 16 19:41:09.046102 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:09.046079    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdz2g_b56eaab7-f00c-48cf-841d-d46f859324e8/extractor/0.log"
Sep 16 19:41:11.468552 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:11.468522    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-75c4c6c467-t8c2x_998302ad-889b-430c-afa8-a60710b4533d/manager/0.log"
Sep 16 19:41:11.530338 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:11.530316    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-dpx5t_8211dde6-b168-4135-a0c4-c4b9a4b28f08/server/0.log"
Sep 16 19:41:11.766234 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:11.766160    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-8x9gn_24075ac7-eed0-47f9-ac7e-227aa61530c2/manager/0.log"
Sep 16 19:41:11.947432 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:11.947407    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-rm98n_9e36f7ba-0e09-4f8b-9b9b-ad936017dd95/seaweedfs/0.log"
Sep 16 19:41:11.971040 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:11.971018    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-jkz2w_96b04e12-4ee6-4e95-b400-a33a933009e8/seaweedfs-tls-custom/0.log"
Sep 16 19:41:12.697681 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:12.697654    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-zvmlb"
Sep 16 19:41:16.660598 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:16.660566    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xmn8h_6678e175-f02f-43f4-9a15-53efd70313a1/kube-storage-version-migrator-operator/1.log"
Sep 16 19:41:16.661836 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:16.661815    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xmn8h_6678e175-f02f-43f4-9a15-53efd70313a1/kube-storage-version-migrator-operator/0.log"
Sep 16 19:41:17.856457 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:17.856379    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj6rj_bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:41:17.880644 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:17.880617    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj6rj_bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226/egress-router-binary-copy/0.log"
Sep 16 19:41:17.905622 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:17.905601    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj6rj_bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226/cni-plugins/0.log"
Sep 16 19:41:17.926933 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:17.926911    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj6rj_bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226/bond-cni-plugin/0.log"
Sep 16 19:41:17.947905 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:17.947876    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj6rj_bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226/routeoverride-cni/0.log"
Sep 16 19:41:17.968192 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:17.968170    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj6rj_bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226/whereabouts-cni-bincopy/0.log"
Sep 16 19:41:17.988024 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:17.987992    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj6rj_bd0c97eb-ddf7-4495-b8bc-02f2eaa8a226/whereabouts-cni/0.log"
Sep 16 19:41:18.240431 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:18.240410    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wh7hw_5f1b6ade-4ed0-4051-8c26-a90a089db3ca/kube-multus/0.log"
Sep 16 19:41:18.306097 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:18.306070    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lcxfx_1f8f4d51-92a4-4bbe-9201-1ec009f2381b/network-metrics-daemon/0.log"
Sep 16 19:41:18.323084 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:18.323063    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lcxfx_1f8f4d51-92a4-4bbe-9201-1ec009f2381b/kube-rbac-proxy/0.log"
Sep 16 19:41:19.385404 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:19.385368    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lpxk7_6764d66c-f9f6-4990-b83c-85216eaae28d/ovn-controller/0.log"
Sep 16 19:41:19.402365 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:19.402344    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lpxk7_6764d66c-f9f6-4990-b83c-85216eaae28d/ovn-acl-logging/0.log"
Sep 16 19:41:19.426341 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:19.426313    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lpxk7_6764d66c-f9f6-4990-b83c-85216eaae28d/kube-rbac-proxy-node/0.log"
Sep 16 19:41:19.446576 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:19.446553    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lpxk7_6764d66c-f9f6-4990-b83c-85216eaae28d/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:41:19.468077 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:19.468059    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lpxk7_6764d66c-f9f6-4990-b83c-85216eaae28d/northd/0.log"
Sep 16 19:41:19.489014 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:19.488994    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lpxk7_6764d66c-f9f6-4990-b83c-85216eaae28d/nbdb/0.log"
Sep 16 19:41:19.519745 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:19.519725    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lpxk7_6764d66c-f9f6-4990-b83c-85216eaae28d/sbdb/0.log"
Sep 16 19:41:19.563568 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:19.563549    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lpxk7_6764d66c-f9f6-4990-b83c-85216eaae28d/ovnkube-controller/0.log"
Sep 16 19:41:20.844053 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:20.844024    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-sbk9q_098cca9f-23f0-4259-954a-fc6a8dbac714/check-endpoints/0.log"
Sep 16 19:41:20.869145 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:20.869110    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-cvrx8_c9f93bb3-6b63-4d1e-bf7e-46e0b3e93123/network-check-target-container/0.log"
Sep 16 19:41:21.844139 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:21.844113    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-k8glm_79908e31-ecc6-4f92-9af8-bd1ed3d16596/iptables-alerter/0.log"
Sep 16 19:41:22.531087 ip-10-0-139-120 kubenswrapper[2584]: I0916 19:41:22.531062    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-g2rsd_c17acc3f-e899-405c-aa12-55aa19fc7b86/tuned/0.log"