Sep 17 21:56:05.289853 ip-10-0-137-232 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 21:56:05.289869 ip-10-0-137-232 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 21:56:05.289880 ip-10-0-137-232 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 21:56:05.290329 ip-10-0-137-232 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 21:56:16.753598 ip-10-0-137-232 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 21:56:16.753619 ip-10-0-137-232 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1609845444c9442988b95de0fae70727 --
Sep 17 21:58:16.643682 ip-10-0-137-232 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 21:58:20.260255 ip-10-0-137-232 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:20.260255 ip-10-0-137-232 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 21:58:20.260255 ip-10-0-137-232 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:20.260255 ip-10-0-137-232 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 21:58:20.260255 ip-10-0-137-232 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:20.260945 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.260342    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 21:58:20.264809 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264795    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.264809 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264808    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.264809 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264812    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264816    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264819    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264822    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264825    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264828    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264831    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264834    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264839    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264843    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264846    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264849    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264856    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264859    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264861    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264864    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264867    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264870    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264879    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264882    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.264912 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264885    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264887    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264890    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264892    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264895    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264897    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264900    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264903    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264905    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264908    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264910    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264913    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264915    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264918    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264921    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264925    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264929    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264932    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264936    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264939    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.265420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264941    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264944    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264947    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264949    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264952    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264954    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264957    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264959    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264962    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264965    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264968    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264970    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264974    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264977    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264980    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264984    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264986    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264989    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264992    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.265924 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264995    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.264997    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265000    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265002    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265005    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265008    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265010    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265014    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265017    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265019    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265021    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265024    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265027    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265029    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265031    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265034    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265036    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265039    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265041    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265044    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.266380 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265046    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265049    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265052    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265054    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265057    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265059    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265062    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265066    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265069    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265071    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265073    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265076    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265078    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265081    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265083    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265086    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265088    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265090    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265765    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265771    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.266896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265774    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265777    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265779    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265782    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265785    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265788    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265790    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265793    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265795    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265798    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265800    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265803    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265805    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265808    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265811    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265813    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265817    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265820    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265823    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.267372 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265826    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265829    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265831    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265834    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265836    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265839    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265842    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265844    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265847    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265849    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265852    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265854    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265857    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265859    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265862    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265864    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265867    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265869    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265871    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265874    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.267935 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265877    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265879    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265881    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265884    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265886    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265889    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265892    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265894    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265897    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265899    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265902    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265905    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265908    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265911    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265914    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265917    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265921    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265924    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265928    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.268421 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265931    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265934    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265937    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265940    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265943    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265946    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265949    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265952    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265954    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265957    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265960    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265962    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265965    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265967    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265970    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265973    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265975    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265979    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265982    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265984    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.268914 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265987    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265990    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265993    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.265996    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266000    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266003    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266005    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266008    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266010    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266013    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266016    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266018    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266021    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266023    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266026    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266028    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266032    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266035    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266038    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.269402 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266391    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266397    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266400    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266403    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266405    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266408    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266411    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266413    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266416    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266419    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266422    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266442    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266445    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266447    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266450    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266453    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266455    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266459    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266462    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266464    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.269881 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266467    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266470    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266472    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266475    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266477    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266480    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266482    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266485    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266488    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266490    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266493    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266496    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266500    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266503    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266505    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266508    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266511    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266513    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266516    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.270357 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266518    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266520    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266523    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266526    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266528    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266531    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266534    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266536    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266539    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266541    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266544    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266547    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266551    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266554    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266557    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266560    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266563    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266565    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266568    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.270892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266570    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266572    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266575    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266577    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266580    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266582    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266585    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266587    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266590    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266592    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266595    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266598    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266602    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266605    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266607    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266610    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266612    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266615    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266618    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.271392 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266620    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266623    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266625    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266628    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266631    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266633    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266636    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266639    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266641    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266644    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266646    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266648    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266651    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266653    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266656    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266658    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266660    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266663    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266665    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266668    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.271921 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266670    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.266672    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266756    2585 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266802    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266811    2585 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266819    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266823    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266826    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266830    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266835    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266838    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266841    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266844    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266848    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266850    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266854    2585 flags.go:64] FLAG: --cgroup-root=""
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266857    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266860    2585 flags.go:64] FLAG: --client-ca-file=""
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266863    2585 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266866    2585 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266870    2585 flags.go:64] FLAG: --cluster-domain=""
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266873    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266876    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266879    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 21:58:20.272482 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266882    2585 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266886    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266889    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266892    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266895    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266898    2585 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266901    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266904    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266908    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266911    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266915    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266918    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266921    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266924    2585 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266926    2585 flags.go:64] FLAG: --enable-server="true"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266929    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266934    2585 flags.go:64] FLAG: --event-burst="100"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266939    2585 flags.go:64] FLAG: --event-qps="50"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266942    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266945    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266948    2585 flags.go:64] FLAG: --eviction-hard=""
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266951    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266954    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266957    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266962    2585 flags.go:64] FLAG: --eviction-soft=""
Sep 17 21:58:20.273124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266965    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266967    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266970    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266974    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266977    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266979    2585 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266982    2585 flags.go:64] FLAG: --feature-gates=""
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266986    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266989    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266992    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266995    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.266998    2585 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267001    2585 flags.go:64] FLAG: --help="false"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267004    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267007    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267010    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267013    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267017    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267020    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267023    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267025    2585 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267028    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267031    2585 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 21:58:20.273901 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267034    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267038    2585 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267041    2585 flags.go:64] FLAG: --kube-reserved=""
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267044    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267046    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267049    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267052    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267055    2585 flags.go:64] FLAG: --lock-file=""
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267059    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267062    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267065    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267070    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267072    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267076    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267079    2585 flags.go:64] FLAG: --logging-format="text"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267081    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267084    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267087    2585 flags.go:64] FLAG: --manifest-url=""
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267090    2585 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267094    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267097    2585 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267101    2585 flags.go:64] FLAG: --max-pods="110"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267104    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267107    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267109    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 21:58:20.274573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267112    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267115    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267118    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267121    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267128    2585 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267131    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267134    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267137    2585 flags.go:64] FLAG: --pod-cidr=""
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267141    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267146    2585 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267149    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267152    2585 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267155    2585 flags.go:64] FLAG: --port="10250"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267158    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267160    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b66f40a0db46206d"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267163    2585 flags.go:64] FLAG: --qos-reserved=""
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267168    2585 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267171    2585 flags.go:64] FLAG: --register-node="true"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267174    2585 flags.go:64] FLAG: --register-with-taints=""
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267177    2585 flags.go:64] FLAG: --registry-burst="10"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267180    2585 flags.go:64] FLAG: --registry-qps="5"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267183    2585 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267186    2585 flags.go:64] FLAG: --reserved-memory=""
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267190    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267192    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 21:58:20.275230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267196    2585 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267198    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267201    2585 flags.go:64] FLAG: --runonce="false"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267204    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267207    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267210    2585 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267212    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267215    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267218    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267221    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267224    2585 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267227    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267230    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267233    2585 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267235    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267239    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267242    2585 flags.go:64] FLAG: --system-cgroups=""
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267245    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267250    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267253    2585 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267255    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267260    2585 flags.go:64] FLAG: --tls-min-version=""
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267262    2585 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267267    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267269    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 21:58:20.275900 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267273    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267275    2585 flags.go:64] FLAG: --v="2"
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267279    2585 flags.go:64] FLAG: --version="false"
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267283    2585 flags.go:64] FLAG: --vmodule=""
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267287    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.267290    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267403    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267408    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267411    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267414    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267416    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267419    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267421    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267437    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267441    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267443    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267446    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267448    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267452    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267456    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267459    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.276859 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267462    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267466    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267468    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267471    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267474    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267476    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267479    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267481    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267484    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267488    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267492    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267495    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267498    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267501    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267504    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267507    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267510    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267513    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267516    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.277478 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267519    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267523    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267525    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267528    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267531    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267534    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267536    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267539    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267542    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267544    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267547    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267549    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267552    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267554    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267559    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267569    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267572    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267574    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267577    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267579    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.277993 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267582    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267584    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267589    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267591    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267594    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267597    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267599    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267602    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267604    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267607    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267610    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267613    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267616    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267618    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267621    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267624    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267626    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267629    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267631    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267634    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.278504 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267637    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267639    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267642    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267644    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267647    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267649    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267653    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267656    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267658    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267661    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267663    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267666    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267669    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267671    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267675    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267678    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267681    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267683    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267686    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267688    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.278999 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267691    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267694    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267696    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267699    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.267702    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.268162    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.275228    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.275243    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.275262    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.275272    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275326    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275331    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275335    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275338    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275342    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.279559 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275344    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275348    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275350    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275353    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275356    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275358    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275361    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275364    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275367    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275370    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275372    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275375    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275377    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275380    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275382    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275387    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275391    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275394    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275397    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.279934 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275400    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275404    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275406    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275409    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275412    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275414    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275417    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275419    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275421    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275439    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275442    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275445    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275447    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275450    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275452    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275455    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275458    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275460    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275463    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275466    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.280387 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275468    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275471    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275474    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275476    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275479    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275481    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275484    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275487    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275490    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275494    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275497    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275500    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275503    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275506    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275509    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275512    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275514    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275518    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275520    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.280892 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275523    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275525    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275528    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275531    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275533    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275536    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275540    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275543    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275546    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275548    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275550    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275553    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275555    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275558    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275561    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275563    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275565    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275568    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275570    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275573    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.281334 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275575    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275578    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275581    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275583    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275586    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275588    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275590    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275593    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275596    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275599    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275601    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275604    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275607    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275609    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275612    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.281868 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275614    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.275619    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275730    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275735    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275738    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275741    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275743    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275746    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275749    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275753    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275757    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275759    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275762    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275765    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:20.282226 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275768    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275771    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275773    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275776    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275778    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275781    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275784    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275787    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275789    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275792    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275795    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275797    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275800    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275803    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275805    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275808    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275810    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275813    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275816    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275818    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:20.282598 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275821    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275823    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275826    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275828    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275831    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275833    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275836    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275838    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275841    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275844    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275846    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275849    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275851    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275854    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275856    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275859    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275861    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275864    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275867    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275869    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:20.283093 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275872    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275874    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275877    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275879    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275882    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275917    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275958    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275964    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275969    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275974    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275988    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275993    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.275997    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.276003    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277447    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277464    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277469    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277472    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277476    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277482    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:20.283651 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277487    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277493    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277497    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277502    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277506    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277511    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277522    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277527    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277531    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277535    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277539    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277543    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277547    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277554    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277561    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277566    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277570    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277577    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277587    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:20.284139 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277592    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277597    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277602    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277606    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277609    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277612    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277615    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:20.277618    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.277625    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.277909    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.279935    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.281481    2585 server.go:1019] "Starting client certificate rotation"
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.281577    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:20.284620 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.281618    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:20.284949 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.283727    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:20.284949 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.284840    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:20.291674 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.291655    2585 log.go:25] "Validated CRI v1 runtime API"
Sep 17 21:58:20.296913 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.296899    2585 log.go:25] "Validated CRI v1 image API"
Sep 17 21:58:20.298183 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.298156    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 21:58:20.301409 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.301387    2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c156941d-7d1d-4217-b846-db0913516eba:/dev/nvme0n1p3 cd3bf4f1-aec7-4cec-a24b-68e0fb86414d:/dev/nvme0n1p4]
Sep 17 21:58:20.301409 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.301403    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 21:58:20.305543 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.305441    2585 manager.go:216] Machine: {Timestamp:2026-09-17 21:58:20.304961998 +0000 UTC m=+2.167215465 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098939 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dc45cf6895ee007f7fa99fe1b984e SystemUUID:ec2dc45c-f689-5ee0-07f7-fa99fe1b984e BootID:16098454-44c9-4429-88b9-5de0fae70727 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:88:64:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:88:64:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:e7:78:e4:e8:5c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 21:58:20.305543 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.305536    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 21:58:20.305657 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.305605    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 21:58:20.307456 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.307416    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 21:58:20.307587 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.307461    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 21:58:20.308606 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.308592    2585 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 21:58:20.308640 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.308611    2585 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 21:58:20.308640 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.308619    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:20.308640 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.308633    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:20.308723 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.308698    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 21:58:20.309820 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.309805    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:20.312466 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.312454    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:20.312603 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.312593    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 21:58:20.312633 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.312622    2585 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 21:58:20.312633 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.312633    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 21:58:20.312686 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.312646    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 21:58:20.312686 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.312654    2585 kubelet.go:398] "Adding apiserver pod source"
Sep 17 21:58:20.312686 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.312663    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 21:58:20.313519 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.313505    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:20.313573 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.313534    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:20.316920 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.316903    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 21:58:20.319011 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.318998    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 21:58:20.319069 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.319027    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 21:58:20.320064 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320054    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 21:58:20.320098 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320070    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 21:58:20.320098 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320076    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 21:58:20.320098 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320083    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 21:58:20.320098 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320091    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 21:58:20.320098 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320096    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 21:58:20.320359 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320110    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 21:58:20.320359 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320115    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 21:58:20.320359 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320122    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 21:58:20.320359 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320127    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 21:58:20.320359 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320135    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 21:58:20.320359 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320140    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 21:58:20.320359 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320173    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 21:58:20.320359 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.320182    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 21:58:20.323965 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.323951    2585 server.go:1268] "Started kubelet"
Sep 17 21:58:20.324107 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.324057    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 21:58:20.324186 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.324130    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 21:58:20.324186 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.324157    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 21:58:20.324832 ip-10-0-137-232 systemd[1]: Started Kubernetes Kubelet.
Sep 17 21:58:20.325365 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.325194    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 21:58:20.325365 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.325316    2585 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 21:58:20.326737 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.326713    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-232.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 21:58:20.326737 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.326717    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 21:58:20.326895 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.326813    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 21:58:20.327927 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.327904    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 21:58:20.331257 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.331239    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 21:58:20.331366 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.331256    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:20.331366 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.331305    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:20.331898 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.331862    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 21:58:20.331898 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.331862    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 21:58:20.331898 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.331888    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 21:58:20.332085 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.331949    2585 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 21:58:20.332085 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.331958    2585 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 21:58:20.332255 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.332241    2585 factory.go:55] Registering systemd factory
Sep 17 21:58:20.332304 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.332261    2585 factory.go:223] Registration of the systemd container factory successfully
Sep 17 21:58:20.332571 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.332550    2585 factory.go:153] Registering CRI-O factory
Sep 17 21:58:20.332571 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.332574    2585 factory.go:223] Registration of the crio container factory successfully
Sep 17 21:58:20.332687 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.332628    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 21:58:20.332687 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.332646    2585 factory.go:103] Registering Raw factory
Sep 17 21:58:20.332687 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.332670    2585 manager.go:1196] Started watching for new ooms in manager
Sep 17 21:58:20.332999 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.332976    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 21:58:20.333103 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.333088    2585 manager.go:349] Starting recovery of all containers
Sep 17 21:58:20.333174 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.333153    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 21:58:20.334207 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.334185    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8s2pp"
Sep 17 21:58:20.339799 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.339780    2585 manager.go:354] Recovery completed
Sep 17 21:58:20.352369 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.352346    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8s2pp"
Sep 17 21:58:20.353165 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.353141    2585 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 17 21:58:20.357783 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.357768    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.359378 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.359361    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.359478 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.359397    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.359478 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.359412    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.360034 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.360019    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 21:58:20.360034 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.360033    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 21:58:20.360138 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.360047    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:20.360138 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.360093    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.360796 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.360781    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.360857 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.360811    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.360857 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.360825    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.362028 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.362016    2585 policy_none.go:49] "None policy: Start"
Sep 17 21:58:20.362069 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.362031    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 21:58:20.362069 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.362041    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 21:58:20.363952 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.363941    2585 policy_none.go:47] "Start"
Sep 17 21:58:20.409099 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.408976    2585 manager.go:337] "Starting Device Plugin manager"
Sep 17 21:58:20.426937 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.409126    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 21:58:20.426937 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.409139    2585 server.go:87] "Starting device plugin registration server"
Sep 17 21:58:20.426937 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.409359    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 21:58:20.426937 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.409371    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 21:58:20.426937 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.409490    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 21:58:20.426937 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.409566    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 21:58:20.426937 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.409574    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 21:58:20.426937 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.410066    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 21:58:20.426937 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.410114    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:20.457083 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.457062    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 21:58:20.458285 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.458270    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 21:58:20.458369 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.458309    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 21:58:20.458369 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.458326    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 21:58:20.458369 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.458358    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 21:58:20.460572 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.460545    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:20.509491 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.509473    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.510232 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.510215    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.510338 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.510245    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.510338 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.510260    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.510338 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.510281    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.519973 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.519956    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.520022 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.519980    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-232.ec2.internal\": node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:20.529829 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.529811    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:20.558961 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.558944    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal"]
Sep 17 21:58:20.559012 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.559001    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.561169 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.561151    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.561246 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.561179    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.561246 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.561194    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.563345 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.563333    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.563483 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.563470    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.563530 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.563520    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.564035 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.564020    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.564081 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.564045    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.564081 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.564059    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.564081 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.564070    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.564178 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.564086    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.564178 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.564098    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.566257 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.566242    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.566328 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.566268    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:20.566968 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.566952    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:20.567062 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.566982    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:20.567062 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.566997    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:20.587966 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.587948    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-232.ec2.internal\" not found" node="ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.592367 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.592351    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-232.ec2.internal\" not found" node="ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.630469 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.630455    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:20.633763 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.633749    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24b489482888411b94d6705aad4c92c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal\" (UID: \"24b489482888411b94d6705aad4c92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.633808 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.633772    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc9ce5716ceb5cf8fa3712948819d326-config\") pod \"kube-apiserver-proxy-ip-10-0-137-232.ec2.internal\" (UID: \"dc9ce5716ceb5cf8fa3712948819d326\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.633808 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.633789    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24b489482888411b94d6705aad4c92c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal\" (UID: \"24b489482888411b94d6705aad4c92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.730703 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.730685    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:20.733956 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.733938    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24b489482888411b94d6705aad4c92c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal\" (UID: \"24b489482888411b94d6705aad4c92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.734001 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.733966    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc9ce5716ceb5cf8fa3712948819d326-config\") pod \"kube-apiserver-proxy-ip-10-0-137-232.ec2.internal\" (UID: \"dc9ce5716ceb5cf8fa3712948819d326\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.734001 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.733982    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24b489482888411b94d6705aad4c92c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal\" (UID: \"24b489482888411b94d6705aad4c92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.734060 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.734025    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24b489482888411b94d6705aad4c92c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal\" (UID: \"24b489482888411b94d6705aad4c92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.734060 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.734039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24b489482888411b94d6705aad4c92c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal\" (UID: \"24b489482888411b94d6705aad4c92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.734060 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.734054    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc9ce5716ceb5cf8fa3712948819d326-config\") pod \"kube-apiserver-proxy-ip-10-0-137-232.ec2.internal\" (UID: \"dc9ce5716ceb5cf8fa3712948819d326\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.831301 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.831272    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:20.891681 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.891658    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.896894 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:20.896860    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:20.931852 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:20.931827    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:21.032278 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:21.032240    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:21.132713 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:21.132659    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:21.233137 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:21.233107    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:21.281414 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.281394    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 21:58:21.281793 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.281549    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:21.334114 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:21.334089    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:21.335522 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.335505    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:21.359010 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.358983    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 21:53:20 +0000 UTC" deadline="2028-05-19 20:22:47.216005032 +0000 UTC"
Sep 17 21:58:21.359010 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.359009    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14638h24m25.856999194s"
Sep 17 21:58:21.434517 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:21.434455    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:21.461848 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.461829    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:21.512888 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:21.512863    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9ce5716ceb5cf8fa3712948819d326.slice/crio-dbc6edcdeebf1fc162be9fa644612a899f1b6a405fb629ee49660906bc84ca1a WatchSource:0}: Error finding container dbc6edcdeebf1fc162be9fa644612a899f1b6a405fb629ee49660906bc84ca1a: Status 404 returned error can't find the container with id dbc6edcdeebf1fc162be9fa644612a899f1b6a405fb629ee49660906bc84ca1a
Sep 17 21:58:21.513420 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:21.513400    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b489482888411b94d6705aad4c92c1.slice/crio-11b6a29f9086fb33c75a9f8382da891cb74c48a50961279cc5810f830d484d6c WatchSource:0}: Error finding container 11b6a29f9086fb33c75a9f8382da891cb74c48a50961279cc5810f830d484d6c: Status 404 returned error can't find the container with id 11b6a29f9086fb33c75a9f8382da891cb74c48a50961279cc5810f830d484d6c
Sep 17 21:58:21.518126 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.518112    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:58:21.534984 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:21.534963    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-232.ec2.internal\" not found"
Sep 17 21:58:21.547902 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.547885    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:21.631769 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.631749    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:21.643091 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.643077    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:21.644193 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.644181    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal"
Sep 17 21:58:21.650827 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:21.650814    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:22.314677 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.314645    2585 apiserver.go:52] "Watching apiserver"
Sep 17 21:58:22.323809 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.323769    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 21:58:22.324118 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.324090    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-9rxzt","openshift-multus/network-metrics-daemon-lm5n6","openshift-network-diagnostics/network-check-target-m2r9c","openshift-network-operator/iptables-alerter-f64cd","kube-system/konnectivity-agent-rfrx9","kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal","openshift-cluster-node-tuning-operator/tuned-j79q4","openshift-image-registry/node-ca-jbrr7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal","openshift-multus/multus-5lmb6","openshift-multus/multus-additional-cni-plugins-qcl8l","openshift-ovn-kubernetes/ovnkube-node-b9ddd","kube-system/global-pull-secret-syncer-tr566","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"]
Sep 17 21:58:22.326562 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.326539    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.329136 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.329108    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bnq92\""
Sep 17 21:58:22.329306 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.329289    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.329306 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.329300    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:22.329443 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.329356    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:22.329443 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.329296    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:22.331273 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.331257    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:22.331363 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.331312    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:22.333021 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.333002    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:22.334523 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.334503    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.336507 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.336415    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 21:58:22.337091 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.336751    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:22.337091 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.336835    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cjw6z\""
Sep 17 21:58:22.337091 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.336955    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:22.337759 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.337527    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:22.339784 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.339763    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rnrjz\""
Sep 17 21:58:22.339953 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.339941    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 21:58:22.340073 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.340061    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 21:58:22.340734 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.340541    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.341793 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341740    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50e0dc15-2515-4770-8887-9643beec1691-host-slash\") pod \"iptables-alerter-f64cd\" (UID: \"50e0dc15-2515-4770-8887-9643beec1691\") " pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.341880 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341802    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-kubernetes\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.341880 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341824    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-run\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.341880 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341839    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-lib-modules\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.341880 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341854    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45d76677-9235-42a6-8963-04df258d406a-kubelet-config\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.341880 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341869    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45d76677-9235-42a6-8963-04df258d406a-dbus\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwjhj\" (UniqueName: \"kubernetes.io/projected/50e0dc15-2515-4770-8887-9643beec1691-kube-api-access-jwjhj\") pod \"iptables-alerter-f64cd\" (UID: \"50e0dc15-2515-4770-8887-9643beec1691\") " pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341896    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-sysconfig\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341913    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-sysctl-d\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341932    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-sysctl-conf\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341945    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-host\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341958    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40867f68-4f1a-49e5-a580-b872ac068cac-etc-tuned\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341970    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40867f68-4f1a-49e5-a580-b872ac068cac-tmp\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.341988    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-modprobe-d\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.342043    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-sys\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.342057    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-var-lib-kubelet\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.342071    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdsrg\" (UniqueName: \"kubernetes.io/projected/40867f68-4f1a-49e5-a580-b872ac068cac-kube-api-access-cdsrg\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.342086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50e0dc15-2515-4770-8887-9643beec1691-iptables-alerter-script\") pod \"iptables-alerter-f64cd\" (UID: \"50e0dc15-2515-4770-8887-9643beec1691\") " pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.342100    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-systemd\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.342115    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.342124 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.342129    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:22.342899 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.342421    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.342899 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.342845    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wg9ms\""
Sep 17 21:58:22.343107 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.343090    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 21:58:22.343166 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.343121    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 21:58:22.343396 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.343317    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 21:58:22.344643 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.344621    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 21:58:22.344742 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.344681    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 21:58:22.344802 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.344759    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.344869 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.344853    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m9t96\""
Sep 17 21:58:22.345086 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.345072    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 21:58:22.347077 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.347056    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j7b9c\""
Sep 17 21:58:22.347439 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.347399    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 21:58:22.347439 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.347415    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 21:58:22.347913 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.347686    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:22.347913 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.347745    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:22.347913 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.347903    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:22.350965 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.350947    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.353684 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.353235    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.353881 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.353864    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 21:58:22.354068 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.354053    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kqwlt\""
Sep 17 21:58:22.354068 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.354063    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 21:58:22.354216 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.354202    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 21:58:22.354411 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.354391    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 21:58:22.355552 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.355533    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.355652 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.355543    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 21:58:22.356772 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.356729    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 21:58:22.359917 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.357800    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qvz9s\""
Sep 17 21:58:22.359917 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.358387    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 21:58:22.359917 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.358778    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 21:58:22.359917 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.359048    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 21:58:22.359917 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.359333    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 21:58:22.359917 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.359725    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 21:58:22.361080 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.361028    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 21:58:22.361080 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.361066    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dgcjl\""
Sep 17 21:58:22.366164 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.366144    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wg5c7"
Sep 17 21:58:22.374537 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.374520    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wg5c7"
Sep 17 21:58:22.433420 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.433359    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 21:58:22.442990 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.442963    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwjhj\" (UniqueName: \"kubernetes.io/projected/50e0dc15-2515-4770-8887-9643beec1691-kube-api-access-jwjhj\") pod \"iptables-alerter-f64cd\" (UID: \"50e0dc15-2515-4770-8887-9643beec1691\") " pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.442990 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.442993    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40867f68-4f1a-49e5-a580-b872ac068cac-etc-tuned\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.443144 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443016    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqf9\" (UniqueName: \"kubernetes.io/projected/ac293f58-ae69-4c22-8804-44545908c47c-kube-api-access-9zqf9\") pod \"node-ca-jbrr7\" (UID: \"ac293f58-ae69-4c22-8804-44545908c47c\") " pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.443144 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443047    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdmx\" (UniqueName: \"kubernetes.io/projected/cdf9f479-ec36-4720-b83f-a0484c14cda4-kube-api-access-8zdmx\") pod \"node-resolver-9rxzt\" (UID: \"cdf9f479-ec36-4720-b83f-a0484c14cda4\") " pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.443144 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443070    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:22.443144 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443089    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5116c2df-4d3f-4892-91a3-b04ffe887787-cnibin\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.443144 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443108    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdsrg\" (UniqueName: \"kubernetes.io/projected/40867f68-4f1a-49e5-a580-b872ac068cac-kube-api-access-cdsrg\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.443144 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-run-openvswitch\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.443144 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443139    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ec01088-cc90-4188-a393-e872a5f5be60-ovnkube-script-lib\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443154    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-etc-kubernetes\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443176    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5116c2df-4d3f-4892-91a3-b04ffe887787-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443196    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443220    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-kubelet\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443242    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/322caaa1-110c-4e1a-96cf-a3d4091e9b56-agent-certs\") pod \"konnectivity-agent-rfrx9\" (UID: \"322caaa1-110c-4e1a-96cf-a3d4091e9b56\") " pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443263    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-var-lib-cni-multus\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443278    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-run\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443292    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-slash\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443316    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443360    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwl7\" (UniqueName: \"kubernetes.io/projected/9ec01088-cc90-4188-a393-e872a5f5be60-kube-api-access-8jwl7\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443382    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac293f58-ae69-4c22-8804-44545908c47c-serviceca\") pod \"node-ca-jbrr7\" (UID: \"ac293f58-ae69-4c22-8804-44545908c47c\") " pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-hostroot\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.443415 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443420    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5116c2df-4d3f-4892-91a3-b04ffe887787-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443458    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-sysconfig\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443488    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40867f68-4f1a-49e5-a580-b872ac068cac-tmp\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443511    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-node-log\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443533    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-sys-fs\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443553    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66ch\" (UniqueName: \"kubernetes.io/projected/3bc8002f-2d01-4adc-8b18-d657102de70a-kube-api-access-p66ch\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443568    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5116c2df-4d3f-4892-91a3-b04ffe887787-system-cni-dir\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443599    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443613    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-modprobe-d\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443958    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-run\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443966    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-sysconfig\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.444002 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.443995    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-var-lib-kubelet\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444038    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-cnibin\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.444066    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444093    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-var-lib-kubelet\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.444128    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret podName:45d76677-9235-42a6-8963-04df258d406a nodeName:}" failed. No retries permitted until 2026-09-17 21:58:22.944108058 +0000 UTC m=+4.806361515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret") pod "global-pull-secret-syncer-tr566" (UID: "45d76677-9235-42a6-8963-04df258d406a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444150    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-daemon-config\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444177    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-systemd-units\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444202    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-cni-netd\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444208    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-modprobe-d\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444223    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ec01088-cc90-4188-a393-e872a5f5be60-env-overrides\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444267    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fxjw\" (UniqueName: \"kubernetes.io/projected/12a3a05c-bf13-4011-a95e-36271b66ec5d-kube-api-access-7fxjw\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50e0dc15-2515-4770-8887-9643beec1691-host-slash\") pod \"iptables-alerter-f64cd\" (UID: \"50e0dc15-2515-4770-8887-9643beec1691\") " pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444333    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-kubernetes\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444348    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50e0dc15-2515-4770-8887-9643beec1691-host-slash\") pod \"iptables-alerter-f64cd\" (UID: \"50e0dc15-2515-4770-8887-9643beec1691\") " pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.444380 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444382    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf9f479-ec36-4720-b83f-a0484c14cda4-tmp-dir\") pod \"node-resolver-9rxzt\" (UID: \"cdf9f479-ec36-4720-b83f-a0484c14cda4\") " pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-cni-dir\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444454    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-sysctl-d\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444475    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-kubernetes\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444508    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-host\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444540    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-run-ovn\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444566    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-registration-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444615    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-sysctl-d\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444624    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-host\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-device-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444696    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68f7b17b-4a7c-4f10-99f0-84652c10602d-cni-binary-copy\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444721    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-socket-dir-parent\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444769    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5116c2df-4d3f-4892-91a3-b04ffe887787-cni-binary-copy\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444806    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-run-systemd\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444841    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ec01088-cc90-4188-a393-e872a5f5be60-ovnkube-config\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444866    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/322caaa1-110c-4e1a-96cf-a3d4091e9b56-konnectivity-ca\") pod \"konnectivity-agent-rfrx9\" (UID: \"322caaa1-110c-4e1a-96cf-a3d4091e9b56\") " pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444891    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5116c2df-4d3f-4892-91a3-b04ffe887787-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.445045 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444919    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpvbg\" (UniqueName: \"kubernetes.io/projected/5116c2df-4d3f-4892-91a3-b04ffe887787-kube-api-access-mpvbg\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.444963    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-run-netns\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445010    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-conf-dir\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445037    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-run-multus-certs\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445060    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d67z2\" (UniqueName: \"kubernetes.io/projected/68f7b17b-4a7c-4f10-99f0-84652c10602d-kube-api-access-d67z2\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445089    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45d76677-9235-42a6-8963-04df258d406a-dbus\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445113    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-cni-bin\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445136    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac293f58-ae69-4c22-8804-44545908c47c-host\") pod \"node-ca-jbrr7\" (UID: \"ac293f58-ae69-4c22-8804-44545908c47c\") " pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445158    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-var-lib-kubelet\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445194    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-sysctl-conf\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445227    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-etc-openvswitch\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445253    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdf9f479-ec36-4720-b83f-a0484c14cda4-hosts-file\") pod \"node-resolver-9rxzt\" (UID: \"cdf9f479-ec36-4720-b83f-a0484c14cda4\") " pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445280    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-sys\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445304    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ec01088-cc90-4188-a393-e872a5f5be60-ovn-node-metrics-cert\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445326    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/45d76677-9235-42a6-8963-04df258d406a-dbus\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445344    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-socket-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445373    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50e0dc15-2515-4770-8887-9643beec1691-iptables-alerter-script\") pod \"iptables-alerter-f64cd\" (UID: \"50e0dc15-2515-4770-8887-9643beec1691\") " pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.445844 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445389    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-sys\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445398    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-systemd\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445402    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-sysctl-conf\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445463    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445482    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-etc-systemd\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445497    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-run-netns\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445514    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-var-lib-openvswitch\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445546    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-run-ovn-kubernetes\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445562    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445576    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-run-k8s-cni-cncf-io\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445592    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-lib-modules\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445634    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45d76677-9235-42a6-8963-04df258d406a-kubelet-config\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445667    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-log-socket\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445693    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445712    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-system-cni-dir\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445726    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-var-lib-cni-bin\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445782    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40867f68-4f1a-49e5-a580-b872ac068cac-lib-modules\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.446536 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445809    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/45d76677-9235-42a6-8963-04df258d406a-kubelet-config\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.447192 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.445935    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50e0dc15-2515-4770-8887-9643beec1691-iptables-alerter-script\") pod \"iptables-alerter-f64cd\" (UID: \"50e0dc15-2515-4770-8887-9643beec1691\") " pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.447192 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.447129    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40867f68-4f1a-49e5-a580-b872ac068cac-tmp\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.447192 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.447144    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40867f68-4f1a-49e5-a580-b872ac068cac-etc-tuned\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.452417 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.452395    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:22.452417 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.452420    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:22.452607 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.452450    2585 projected.go:196] Error preparing data for projected volume kube-api-access-f5jkh for pod openshift-network-diagnostics/network-check-target-m2r9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:22.452607 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.452506    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh podName:e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb nodeName:}" failed. No retries permitted until 2026-09-17 21:58:22.952489405 +0000 UTC m=+4.814742860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f5jkh" (UniqueName: "kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh") pod "network-check-target-m2r9c" (UID: "e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:22.453036 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.452915    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdsrg\" (UniqueName: \"kubernetes.io/projected/40867f68-4f1a-49e5-a580-b872ac068cac-kube-api-access-cdsrg\") pod \"tuned-j79q4\" (UID: \"40867f68-4f1a-49e5-a580-b872ac068cac\") " pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.454010 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.453983    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwjhj\" (UniqueName: \"kubernetes.io/projected/50e0dc15-2515-4770-8887-9643beec1691-kube-api-access-jwjhj\") pod \"iptables-alerter-f64cd\" (UID: \"50e0dc15-2515-4770-8887-9643beec1691\") " pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.462656 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.462608    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal" event={"ID":"dc9ce5716ceb5cf8fa3712948819d326","Type":"ContainerStarted","Data":"dbc6edcdeebf1fc162be9fa644612a899f1b6a405fb629ee49660906bc84ca1a"}
Sep 17 21:58:22.463937 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.463913    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal" event={"ID":"24b489482888411b94d6705aad4c92c1","Type":"ContainerStarted","Data":"11b6a29f9086fb33c75a9f8382da891cb74c48a50961279cc5810f830d484d6c"}
Sep 17 21:58:22.537583 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.537550    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:22.546502 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546463    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqf9\" (UniqueName: \"kubernetes.io/projected/ac293f58-ae69-4c22-8804-44545908c47c-kube-api-access-9zqf9\") pod \"node-ca-jbrr7\" (UID: \"ac293f58-ae69-4c22-8804-44545908c47c\") " pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.546671 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdmx\" (UniqueName: \"kubernetes.io/projected/cdf9f479-ec36-4720-b83f-a0484c14cda4-kube-api-access-8zdmx\") pod \"node-resolver-9rxzt\" (UID: \"cdf9f479-ec36-4720-b83f-a0484c14cda4\") " pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.546671 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546540    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:22.546671 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546565    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5116c2df-4d3f-4892-91a3-b04ffe887787-cnibin\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.546671 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546590    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-run-openvswitch\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.546671 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546613    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ec01088-cc90-4188-a393-e872a5f5be60-ovnkube-script-lib\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.546671 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546636    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-etc-kubernetes\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.546671 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546650    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5116c2df-4d3f-4892-91a3-b04ffe887787-cnibin\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.546671 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546662    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5116c2df-4d3f-4892-91a3-b04ffe887787-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.546708    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546719    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-kubelet\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546744    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/322caaa1-110c-4e1a-96cf-a3d4091e9b56-agent-certs\") pod \"konnectivity-agent-rfrx9\" (UID: \"322caaa1-110c-4e1a-96cf-a3d4091e9b56\") " pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546768    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-var-lib-cni-multus\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546776    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-kubelet\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.546796    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs podName:12a3a05c-bf13-4011-a95e-36271b66ec5d nodeName:}" failed. No retries permitted until 2026-09-17 21:58:23.04676778 +0000 UTC m=+4.909021249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs") pod "network-metrics-daemon-lm5n6" (UID: "12a3a05c-bf13-4011-a95e-36271b66ec5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546813    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-var-lib-cni-multus\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-slash\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546869    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwl7\" (UniqueName: \"kubernetes.io/projected/9ec01088-cc90-4188-a393-e872a5f5be60-kube-api-access-8jwl7\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546928    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac293f58-ae69-4c22-8804-44545908c47c-serviceca\") pod \"node-ca-jbrr7\" (UID: \"ac293f58-ae69-4c22-8804-44545908c47c\") " pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546952    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-hostroot\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.546981    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5116c2df-4d3f-4892-91a3-b04ffe887787-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547010    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-node-log\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547014    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-etc-kubernetes\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547035    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-sys-fs\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.547070 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547063    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66ch\" (UniqueName: \"kubernetes.io/projected/3bc8002f-2d01-4adc-8b18-d657102de70a-kube-api-access-p66ch\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547075    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-run-openvswitch\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547088    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5116c2df-4d3f-4892-91a3-b04ffe887787-system-cni-dir\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547117    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-cnibin\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547121    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-hostroot\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547127    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5116c2df-4d3f-4892-91a3-b04ffe887787-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547143    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-daemon-config\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547159    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-slash\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547180    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547170    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-systemd-units\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547225    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-systemd-units\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-cni-netd\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547255    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-cni-netd\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547270    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ec01088-cc90-4188-a393-e872a5f5be60-env-overrides\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547298    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5116c2df-4d3f-4892-91a3-b04ffe887787-system-cni-dir\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547299    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxjw\" (UniqueName: \"kubernetes.io/projected/12a3a05c-bf13-4011-a95e-36271b66ec5d-kube-api-access-7fxjw\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547333    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf9f479-ec36-4720-b83f-a0484c14cda4-tmp-dir\") pod \"node-resolver-9rxzt\" (UID: \"cdf9f479-ec36-4720-b83f-a0484c14cda4\") " pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.547906 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547333    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-node-log\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547356    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-cni-dir\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547385    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-run-ovn\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547408    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-registration-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547460    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-device-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547490    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68f7b17b-4a7c-4f10-99f0-84652c10602d-cni-binary-copy\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547516    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-socket-dir-parent\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547569    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-cni-dir\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547577    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-cnibin\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547614    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac293f58-ae69-4c22-8804-44545908c47c-serviceca\") pod \"node-ca-jbrr7\" (UID: \"ac293f58-ae69-4c22-8804-44545908c47c\") " pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547627    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-sys-fs\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547638    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5116c2df-4d3f-4892-91a3-b04ffe887787-cni-binary-copy\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547728    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-run-systemd\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547751    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ec01088-cc90-4188-a393-e872a5f5be60-ovnkube-config\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547774    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/322caaa1-110c-4e1a-96cf-a3d4091e9b56-konnectivity-ca\") pod \"konnectivity-agent-rfrx9\" (UID: \"322caaa1-110c-4e1a-96cf-a3d4091e9b56\") " pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5116c2df-4d3f-4892-91a3-b04ffe887787-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547798    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5116c2df-4d3f-4892-91a3-b04ffe887787-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.548754 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547824    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvbg\" (UniqueName: \"kubernetes.io/projected/5116c2df-4d3f-4892-91a3-b04ffe887787-kube-api-access-mpvbg\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547846    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-socket-dir-parent\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547850    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-run-netns\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547869    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-run-ovn\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-conf-dir\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547878    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-registration-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547893    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf9f479-ec36-4720-b83f-a0484c14cda4-tmp-dir\") pod \"node-resolver-9rxzt\" (UID: \"cdf9f479-ec36-4720-b83f-a0484c14cda4\") " pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547905    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-device-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547513    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ec01088-cc90-4188-a393-e872a5f5be60-ovnkube-script-lib\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547912    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-run-multus-certs\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547958    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-run-multus-certs\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.547984    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d67z2\" (UniqueName: \"kubernetes.io/projected/68f7b17b-4a7c-4f10-99f0-84652c10602d-kube-api-access-d67z2\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548007    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-cni-bin\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548027    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac293f58-ae69-4c22-8804-44545908c47c-host\") pod \"node-ca-jbrr7\" (UID: \"ac293f58-ae69-4c22-8804-44545908c47c\") " pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548049    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-var-lib-kubelet\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548069    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5116c2df-4d3f-4892-91a3-b04ffe887787-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548095    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-etc-openvswitch\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548114    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-run-systemd\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.549488 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548120    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdf9f479-ec36-4720-b83f-a0484c14cda4-hosts-file\") pod \"node-resolver-9rxzt\" (UID: \"cdf9f479-ec36-4720-b83f-a0484c14cda4\") " pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548145    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ec01088-cc90-4188-a393-e872a5f5be60-ovn-node-metrics-cert\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548168    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-socket-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548172    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-daemon-config\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548205    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-run-netns\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548212    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68f7b17b-4a7c-4f10-99f0-84652c10602d-cni-binary-copy\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548229    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-var-lib-openvswitch\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548247    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-var-lib-kubelet\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548255    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-run-ovn-kubernetes\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548279    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548302    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-run-k8s-cni-cncf-io\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548322    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-run-netns\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548331    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-log-socket\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548368    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5116c2df-4d3f-4892-91a3-b04ffe887787-cni-binary-copy\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548387    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548403    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-log-socket\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548422    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-system-cni-dir\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548459    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-multus-conf-dir\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548376    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-etc-openvswitch\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548510    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cdf9f479-ec36-4720-b83f-a0484c14cda4-hosts-file\") pod \"node-resolver-9rxzt\" (UID: \"cdf9f479-ec36-4720-b83f-a0484c14cda4\") " pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548526    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-system-cni-dir\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548550    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-var-lib-cni-bin\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548557    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ec01088-cc90-4188-a393-e872a5f5be60-ovnkube-config\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548604    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-socket-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548621    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-run-ovn-kubernetes\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548625    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-run-netns\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548629    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548639    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3bc8002f-2d01-4adc-8b18-d657102de70a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548649    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-host-cni-bin\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548663    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ec01088-cc90-4188-a393-e872a5f5be60-var-lib-openvswitch\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548673    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac293f58-ae69-4c22-8804-44545908c47c-host\") pod \"node-ca-jbrr7\" (UID: \"ac293f58-ae69-4c22-8804-44545908c47c\") " pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548680    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-var-lib-cni-bin\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.548697    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/68f7b17b-4a7c-4f10-99f0-84652c10602d-host-run-k8s-cni-cncf-io\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.549039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ec01088-cc90-4188-a393-e872a5f5be60-env-overrides\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.549136    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/322caaa1-110c-4e1a-96cf-a3d4091e9b56-konnectivity-ca\") pod \"konnectivity-agent-rfrx9\" (UID: \"322caaa1-110c-4e1a-96cf-a3d4091e9b56\") " pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:22.550863 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.550006    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/322caaa1-110c-4e1a-96cf-a3d4091e9b56-agent-certs\") pod \"konnectivity-agent-rfrx9\" (UID: \"322caaa1-110c-4e1a-96cf-a3d4091e9b56\") " pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:22.551631 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.550877    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ec01088-cc90-4188-a393-e872a5f5be60-ovn-node-metrics-cert\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.558598 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.558450    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqf9\" (UniqueName: \"kubernetes.io/projected/ac293f58-ae69-4c22-8804-44545908c47c-kube-api-access-9zqf9\") pod \"node-ca-jbrr7\" (UID: \"ac293f58-ae69-4c22-8804-44545908c47c\") " pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.558598 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.558522    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdmx\" (UniqueName: \"kubernetes.io/projected/cdf9f479-ec36-4720-b83f-a0484c14cda4-kube-api-access-8zdmx\") pod \"node-resolver-9rxzt\" (UID: \"cdf9f479-ec36-4720-b83f-a0484c14cda4\") " pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.558598 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.558565    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxjw\" (UniqueName: \"kubernetes.io/projected/12a3a05c-bf13-4011-a95e-36271b66ec5d-kube-api-access-7fxjw\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:22.559147 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.559128    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66ch\" (UniqueName: \"kubernetes.io/projected/3bc8002f-2d01-4adc-8b18-d657102de70a-kube-api-access-p66ch\") pod \"aws-ebs-csi-driver-node-kbwcn\" (UID: \"3bc8002f-2d01-4adc-8b18-d657102de70a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.559812 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.559776    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d67z2\" (UniqueName: \"kubernetes.io/projected/68f7b17b-4a7c-4f10-99f0-84652c10602d-kube-api-access-d67z2\") pod \"multus-5lmb6\" (UID: \"68f7b17b-4a7c-4f10-99f0-84652c10602d\") " pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.560130 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.560103    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwl7\" (UniqueName: \"kubernetes.io/projected/9ec01088-cc90-4188-a393-e872a5f5be60-kube-api-access-8jwl7\") pod \"ovnkube-node-b9ddd\" (UID: \"9ec01088-cc90-4188-a393-e872a5f5be60\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.560198 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.560172    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpvbg\" (UniqueName: \"kubernetes.io/projected/5116c2df-4d3f-4892-91a3-b04ffe887787-kube-api-access-mpvbg\") pod \"multus-additional-cni-plugins-qcl8l\" (UID: \"5116c2df-4d3f-4892-91a3-b04ffe887787\") " pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.637229 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.637191    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j79q4"
Sep 17 21:58:22.648778 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.648754    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f64cd"
Sep 17 21:58:22.661394 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.661369    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:22.665052 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.665031    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn"
Sep 17 21:58:22.671769 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.671738    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbrr7"
Sep 17 21:58:22.679455 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.679437    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9rxzt"
Sep 17 21:58:22.688088 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.688047    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5lmb6"
Sep 17 21:58:22.695557 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.695544    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qcl8l"
Sep 17 21:58:22.703197 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.703178    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:22.950499 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:22.950409    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:22.950651 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.950558    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:22.950651 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:22.950625    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret podName:45d76677-9235-42a6-8963-04df258d406a nodeName:}" failed. No retries permitted until 2026-09-17 21:58:23.95060932 +0000 UTC m=+5.812862794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret") pod "global-pull-secret-syncer-tr566" (UID: "45d76677-9235-42a6-8963-04df258d406a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:23.051634 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.051599    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:23.051804 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.051665    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:23.051804 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:23.051784    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:23.051918 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:23.051817    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:23.051918 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:23.051833    2585 projected.go:196] Error preparing data for projected volume kube-api-access-f5jkh for pod openshift-network-diagnostics/network-check-target-m2r9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:23.051918 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:23.051784    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:23.051918 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:23.051889    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh podName:e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb nodeName:}" failed. No retries permitted until 2026-09-17 21:58:24.051871913 +0000 UTC m=+5.914125381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f5jkh" (UniqueName: "kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh") pod "network-check-target-m2r9c" (UID: "e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:23.052111 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:23.051932    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs podName:12a3a05c-bf13-4011-a95e-36271b66ec5d nodeName:}" failed. No retries permitted until 2026-09-17 21:58:24.051915815 +0000 UTC m=+5.914169269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs") pod "network-metrics-daemon-lm5n6" (UID: "12a3a05c-bf13-4011-a95e-36271b66ec5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:23.377501 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.377467    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:22 +0000 UTC" deadline="2028-02-24 19:00:18.295027538 +0000 UTC"
Sep 17 21:58:23.377501 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.377498    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12597h1m54.917534739s"
Sep 17 21:58:23.387957 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:23.386753    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac293f58_ae69_4c22_8804_44545908c47c.slice/crio-d35a0e59a3ebd89de95ecb59a16f9d3dd355116909de6c5abec7b76730e0a4cc WatchSource:0}: Error finding container d35a0e59a3ebd89de95ecb59a16f9d3dd355116909de6c5abec7b76730e0a4cc: Status 404 returned error can't find the container with id d35a0e59a3ebd89de95ecb59a16f9d3dd355116909de6c5abec7b76730e0a4cc
Sep 17 21:58:23.392012 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:23.391984    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e0dc15_2515_4770_8887_9643beec1691.slice/crio-16eea0e8c2d1b127f3736b5f5c3f206b1a5dab964fdcadc5a63b82ef3a6aa7d4 WatchSource:0}: Error finding container 16eea0e8c2d1b127f3736b5f5c3f206b1a5dab964fdcadc5a63b82ef3a6aa7d4: Status 404 returned error can't find the container with id 16eea0e8c2d1b127f3736b5f5c3f206b1a5dab964fdcadc5a63b82ef3a6aa7d4
Sep 17 21:58:23.394131 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:23.394105    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322caaa1_110c_4e1a_96cf_a3d4091e9b56.slice/crio-39fcc6e3f9435fbb45914571d0c26cad6546e361ca67c001ede31169f8acf11b WatchSource:0}: Error finding container 39fcc6e3f9435fbb45914571d0c26cad6546e361ca67c001ede31169f8acf11b: Status 404 returned error can't find the container with id 39fcc6e3f9435fbb45914571d0c26cad6546e361ca67c001ede31169f8acf11b
Sep 17 21:58:23.395473 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:23.395403    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f7b17b_4a7c_4f10_99f0_84652c10602d.slice/crio-32014a983128f07d5b1115ab807ce0c6488bfbc5a7e136ecd970b1337cf43785 WatchSource:0}: Error finding container 32014a983128f07d5b1115ab807ce0c6488bfbc5a7e136ecd970b1337cf43785: Status 404 returned error can't find the container with id 32014a983128f07d5b1115ab807ce0c6488bfbc5a7e136ecd970b1337cf43785
Sep 17 21:58:23.396896 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:23.396863    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40867f68_4f1a_49e5_a580_b872ac068cac.slice/crio-05aa095b2cd1be7ff01cce30e46344fb123b8a963d3fd3241a313b919f7337f4 WatchSource:0}: Error finding container 05aa095b2cd1be7ff01cce30e46344fb123b8a963d3fd3241a313b919f7337f4: Status 404 returned error can't find the container with id 05aa095b2cd1be7ff01cce30e46344fb123b8a963d3fd3241a313b919f7337f4
Sep 17 21:58:23.399389 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:23.397810    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf9f479_ec36_4720_b83f_a0484c14cda4.slice/crio-39399391c025fc07c6ee21e0f3c1aef15bc4fadbf8bc2810bb8524f0d8c07fcc WatchSource:0}: Error finding container 39399391c025fc07c6ee21e0f3c1aef15bc4fadbf8bc2810bb8524f0d8c07fcc: Status 404 returned error can't find the container with id 39399391c025fc07c6ee21e0f3c1aef15bc4fadbf8bc2810bb8524f0d8c07fcc
Sep 17 21:58:23.399389 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:23.398714    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5116c2df_4d3f_4892_91a3_b04ffe887787.slice/crio-bdc595323bb796bea614ab38a81baae22474f3d2bdd0122b297146bdc6a27ada WatchSource:0}: Error finding container bdc595323bb796bea614ab38a81baae22474f3d2bdd0122b297146bdc6a27ada: Status 404 returned error can't find the container with id bdc595323bb796bea614ab38a81baae22474f3d2bdd0122b297146bdc6a27ada
Sep 17 21:58:23.467266 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.467119    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal" event={"ID":"dc9ce5716ceb5cf8fa3712948819d326","Type":"ContainerStarted","Data":"2fd672ee3fd43979d0a880bda429ab343e28d565a116f76441be59373216a90f"}
Sep 17 21:58:23.469364 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.469341    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9rxzt" event={"ID":"cdf9f479-ec36-4720-b83f-a0484c14cda4","Type":"ContainerStarted","Data":"39399391c025fc07c6ee21e0f3c1aef15bc4fadbf8bc2810bb8524f0d8c07fcc"}
Sep 17 21:58:23.472311 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.472290    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5lmb6" event={"ID":"68f7b17b-4a7c-4f10-99f0-84652c10602d","Type":"ContainerStarted","Data":"32014a983128f07d5b1115ab807ce0c6488bfbc5a7e136ecd970b1337cf43785"}
Sep 17 21:58:23.473803 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.473785    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rfrx9" event={"ID":"322caaa1-110c-4e1a-96cf-a3d4091e9b56","Type":"ContainerStarted","Data":"39fcc6e3f9435fbb45914571d0c26cad6546e361ca67c001ede31169f8acf11b"}
Sep 17 21:58:23.476237 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.476216    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbrr7" event={"ID":"ac293f58-ae69-4c22-8804-44545908c47c","Type":"ContainerStarted","Data":"d35a0e59a3ebd89de95ecb59a16f9d3dd355116909de6c5abec7b76730e0a4cc"}
Sep 17 21:58:23.480101 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.480081    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerStarted","Data":"6b7e6a0e73f7bb2fb6f438bb4f12c36ccbce50b6b0b8cc8dedcb1a9d90aeb97f"}
Sep 17 21:58:23.482635 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.482597    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-232.ec2.internal" podStartSLOduration=2.482581921 podStartE2EDuration="2.482581921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:23.482158594 +0000 UTC m=+5.344412071" watchObservedRunningTime="2026-09-17 21:58:23.482581921 +0000 UTC m=+5.344835398"
Sep 17 21:58:23.484569 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.484548    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcl8l" event={"ID":"5116c2df-4d3f-4892-91a3-b04ffe887787","Type":"ContainerStarted","Data":"bdc595323bb796bea614ab38a81baae22474f3d2bdd0122b297146bdc6a27ada"}
Sep 17 21:58:23.487805 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.487785    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j79q4" event={"ID":"40867f68-4f1a-49e5-a580-b872ac068cac","Type":"ContainerStarted","Data":"05aa095b2cd1be7ff01cce30e46344fb123b8a963d3fd3241a313b919f7337f4"}
Sep 17 21:58:23.489154 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.489128    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f64cd" event={"ID":"50e0dc15-2515-4770-8887-9643beec1691","Type":"ContainerStarted","Data":"16eea0e8c2d1b127f3736b5f5c3f206b1a5dab964fdcadc5a63b82ef3a6aa7d4"}
Sep 17 21:58:23.490315 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.490291    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn" event={"ID":"3bc8002f-2d01-4adc-8b18-d657102de70a","Type":"ContainerStarted","Data":"004cbe392986ddc91558fd683ddf1001e7bd75f1dd317862f624f27d778f0c20"}
Sep 17 21:58:23.958478 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:23.958309    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:23.958637 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:23.958519    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:23.958637 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:23.958580    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret podName:45d76677-9235-42a6-8963-04df258d406a nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.958562915 +0000 UTC m=+7.820816392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret") pod "global-pull-secret-syncer-tr566" (UID: "45d76677-9235-42a6-8963-04df258d406a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:24.059943 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:24.059660    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:24.059943 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:24.059721    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:24.059943 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:24.059887    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:24.059943 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:24.059948    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs podName:12a3a05c-bf13-4011-a95e-36271b66ec5d nodeName:}" failed. No retries permitted until 2026-09-17 21:58:26.059928756 +0000 UTC m=+7.922182215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs") pod "network-metrics-daemon-lm5n6" (UID: "12a3a05c-bf13-4011-a95e-36271b66ec5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:24.060367 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:24.060337    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:24.060367 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:24.060361    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:24.060548 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:24.060373    2585 projected.go:196] Error preparing data for projected volume kube-api-access-f5jkh for pod openshift-network-diagnostics/network-check-target-m2r9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:24.060548 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:24.060416    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh podName:e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb nodeName:}" failed. No retries permitted until 2026-09-17 21:58:26.0604022 +0000 UTC m=+7.922655656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f5jkh" (UniqueName: "kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh") pod "network-check-target-m2r9c" (UID: "e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:24.378130 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:24.378048    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:22 +0000 UTC" deadline="2028-04-23 14:30:34.897451775 +0000 UTC"
Sep 17 21:58:24.378130 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:24.378101    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14008h32m10.519354323s"
Sep 17 21:58:24.468362 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:24.468337    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:24.468540 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:24.468470    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:24.468872 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:24.468832    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:24.468977 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:24.468947    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:24.468977 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:24.468831    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:24.469085 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:24.469030    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:24.518638 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:24.518594    2585 generic.go:356] "Generic (PLEG): container finished" podID="24b489482888411b94d6705aad4c92c1" containerID="1ba4b82fbd8fc7fbcdc8153e63bdeae2a8eba9631389d62c0e2ae2c78e9ab01f" exitCode=0
Sep 17 21:58:24.518771 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:24.518709    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal" event={"ID":"24b489482888411b94d6705aad4c92c1","Type":"ContainerDied","Data":"1ba4b82fbd8fc7fbcdc8153e63bdeae2a8eba9631389d62c0e2ae2c78e9ab01f"}
Sep 17 21:58:25.534248 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:25.534212    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal" event={"ID":"24b489482888411b94d6705aad4c92c1","Type":"ContainerStarted","Data":"c9a565e45b95440414a7228bf50921071ecdf83aa091d3a485d1cabf5b5dbb02"}
Sep 17 21:58:25.974739 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:25.974656    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:25.974919 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:25.974799    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:25.974919 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:25.974861    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret podName:45d76677-9235-42a6-8963-04df258d406a nodeName:}" failed. No retries permitted until 2026-09-17 21:58:29.974840117 +0000 UTC m=+11.837093586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret") pod "global-pull-secret-syncer-tr566" (UID: "45d76677-9235-42a6-8963-04df258d406a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:26.075088 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:26.075050    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:26.075280 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:26.075102    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:26.075280 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:26.075225    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:26.075280 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:26.075276    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs podName:12a3a05c-bf13-4011-a95e-36271b66ec5d nodeName:}" failed. No retries permitted until 2026-09-17 21:58:30.075260614 +0000 UTC m=+11.937514083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs") pod "network-metrics-daemon-lm5n6" (UID: "12a3a05c-bf13-4011-a95e-36271b66ec5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:26.075710 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:26.075607    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:26.075710 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:26.075627    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:26.075710 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:26.075637    2585 projected.go:196] Error preparing data for projected volume kube-api-access-f5jkh for pod openshift-network-diagnostics/network-check-target-m2r9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:26.075710 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:26.075667    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh podName:e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb nodeName:}" failed. No retries permitted until 2026-09-17 21:58:30.075657217 +0000 UTC m=+11.937910671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f5jkh" (UniqueName: "kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh") pod "network-check-target-m2r9c" (UID: "e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:26.463467 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:26.463416    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:26.463641 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:26.463558    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:26.464227 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:26.463941    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:26.464227 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:26.464055    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:26.464227 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:26.464108    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:26.464227 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:26.464178    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:28.463090 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:28.462655    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:28.463090 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:28.462691    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:28.463090 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:28.462777    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:28.463090 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:28.462818    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:28.463090 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:28.462868    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:28.463090 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:28.462949    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:30.014839 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:30.014799    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:30.015274 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.014995    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:30.015274 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.015073    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret podName:45d76677-9235-42a6-8963-04df258d406a nodeName:}" failed. No retries permitted until 2026-09-17 21:58:38.015054057 +0000 UTC m=+19.877307516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret") pod "global-pull-secret-syncer-tr566" (UID: "45d76677-9235-42a6-8963-04df258d406a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:30.116146 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:30.116111    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:30.116323 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:30.116193    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:30.116387 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.116335    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:30.116387 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.116353    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:30.116387 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.116366    2585 projected.go:196] Error preparing data for projected volume kube-api-access-f5jkh for pod openshift-network-diagnostics/network-check-target-m2r9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:30.116563 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.116420    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh podName:e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb nodeName:}" failed. No retries permitted until 2026-09-17 21:58:38.116402536 +0000 UTC m=+19.978656010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f5jkh" (UniqueName: "kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh") pod "network-check-target-m2r9c" (UID: "e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:30.116928 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.116823    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:30.116928 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.116877    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs podName:12a3a05c-bf13-4011-a95e-36271b66ec5d nodeName:}" failed. No retries permitted until 2026-09-17 21:58:38.116861568 +0000 UTC m=+19.979115026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs") pod "network-metrics-daemon-lm5n6" (UID: "12a3a05c-bf13-4011-a95e-36271b66ec5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:30.459838 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:30.459411    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:30.459838 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.459524    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:30.459838 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:30.459550    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:30.459838 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.459667    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:30.459838 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:30.459726    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:30.459838 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:30.459823    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:32.458607 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:32.458579    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:32.458607 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:32.458592    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:32.459157 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:32.458709    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:32.459157 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:32.458784    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:32.459157 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:32.458777    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:32.459157 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:32.459070    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:34.459182 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:34.459147    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:34.459182 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:34.459177    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:34.459736 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:34.459274    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:34.459736 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:34.459351    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:34.459736 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:34.459380    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:34.459736 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:34.459488    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:36.459052 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:36.459018    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:36.459514 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:36.459063    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:36.459514 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:36.459029    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:36.459514 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:36.459169    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:36.459514 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:36.459322    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:36.459514 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:36.459400    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:38.077028 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:38.076979    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:38.077528 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.077144    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:38.077528 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.077225    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret podName:45d76677-9235-42a6-8963-04df258d406a nodeName:}" failed. No retries permitted until 2026-09-17 21:58:54.077203992 +0000 UTC m=+35.939457468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret") pod "global-pull-secret-syncer-tr566" (UID: "45d76677-9235-42a6-8963-04df258d406a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:38.177610 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:38.177578    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:38.177750 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:38.177620    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:38.177750 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.177717    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:38.177750 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.177734    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:38.177853 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.177753    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:38.177853 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.177763    2585 projected.go:196] Error preparing data for projected volume kube-api-access-f5jkh for pod openshift-network-diagnostics/network-check-target-m2r9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:38.177853 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.177767    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs podName:12a3a05c-bf13-4011-a95e-36271b66ec5d nodeName:}" failed. No retries permitted until 2026-09-17 21:58:54.177752492 +0000 UTC m=+36.040005946 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs") pod "network-metrics-daemon-lm5n6" (UID: "12a3a05c-bf13-4011-a95e-36271b66ec5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:38.177853 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.177801    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh podName:e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb nodeName:}" failed. No retries permitted until 2026-09-17 21:58:54.177789821 +0000 UTC m=+36.040043275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f5jkh" (UniqueName: "kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh") pod "network-check-target-m2r9c" (UID: "e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:38.458706 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:38.458619    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:38.458881 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:38.458619    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:38.458881 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.458749    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:38.458881 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:38.458775    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:38.458881 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.458872    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:38.459060 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:38.458996    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:40.459517 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:40.459485    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:40.459950 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:40.459573    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:40.459950 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:40.459588    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:40.459950 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:40.459668    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:40.459950 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:40.459667    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:40.459950 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:40.459784    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:41.566326 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:41.566099    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rfrx9" event={"ID":"322caaa1-110c-4e1a-96cf-a3d4091e9b56","Type":"ContainerStarted","Data":"581f93d31b6009aa4eca4e399d75f9f251551f52ce46ed9e047d55fa8774c3c9"}
Sep 17 21:58:41.568176 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:41.568125    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbrr7" event={"ID":"ac293f58-ae69-4c22-8804-44545908c47c","Type":"ContainerStarted","Data":"4c111349c8a06fa2889bf0c254a0c2daba0516a0f0103569fecc625f2699a850"}
Sep 17 21:58:41.569779 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:41.569757    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerStarted","Data":"ac83ab78b6edf7f119c385918256f99e067a662e5add4aa5387792857616b1ad"}
Sep 17 21:58:41.571462 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:41.571415    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j79q4" event={"ID":"40867f68-4f1a-49e5-a580-b872ac068cac","Type":"ContainerStarted","Data":"3a792d391917cba1327abe20e5c51c0d564ef79c00c7989f2e0bd93d07a6ab2c"}
Sep 17 21:58:41.572947 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:41.572929    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn" event={"ID":"3bc8002f-2d01-4adc-8b18-d657102de70a","Type":"ContainerStarted","Data":"3889e7f8592806cdf5d8cfc480a9701ee1d609fc1e56cd8468e7ed5ca9da5fce"}
Sep 17 21:58:41.581185 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:41.581139    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rfrx9" podStartSLOduration=3.777465904 podStartE2EDuration="21.581123158s" totalImagesPullingTime="17.803657254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:41.580744275 +0000 UTC m=+23.442997750" watchObservedRunningTime="2026-09-17 21:58:41.581123158 +0000 UTC m=+23.443376636"
Sep 17 21:58:41.581411 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:41.581384    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-232.ec2.internal" podStartSLOduration=20.581374805 podStartE2EDuration="20.581374805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:25.549646586 +0000 UTC m=+7.411900063" watchObservedRunningTime="2026-09-17 21:58:41.581374805 +0000 UTC m=+23.443628284"
Sep 17 21:58:41.595905 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:41.595870    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jbrr7" podStartSLOduration=3.787215648 podStartE2EDuration="21.595858441s" totalImagesPullingTime="17.808642793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:41.595698842 +0000 UTC m=+23.457952317" watchObservedRunningTime="2026-09-17 21:58:41.595858441 +0000 UTC m=+23.458111916"
Sep 17 21:58:41.610602 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:41.610565    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j79q4" podStartSLOduration=3.771499332 podStartE2EDuration="21.610554018s" totalImagesPullingTime="17.839054686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:41.610467409 +0000 UTC m=+23.472720887" watchObservedRunningTime="2026-09-17 21:58:41.610554018 +0000 UTC m=+23.472807495"
Sep 17 21:58:42.175837 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.175802    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:42.176396 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.176377    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:42.459302 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.459275    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:42.459302 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.459297    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:42.459540 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.459275    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:42.459540 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:42.459419    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:42.459540 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:42.459502    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:42.459672 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:42.459581    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:42.576851 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.576822    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9rxzt" event={"ID":"cdf9f479-ec36-4720-b83f-a0484c14cda4","Type":"ContainerStarted","Data":"b864374b44618b27b36731656889204bde065dc208da02aae3c89937eb98bb36"}
Sep 17 21:58:42.578241 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.578220    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5lmb6" event={"ID":"68f7b17b-4a7c-4f10-99f0-84652c10602d","Type":"ContainerStarted","Data":"f8353b76a9cabba56f6bec0610b8ee3266ba6f3b9e2dcbddedda6544a6573290"}
Sep 17 21:58:42.580887 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.580868    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 21:58:42.581214 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.581197    2585 generic.go:356] "Generic (PLEG): container finished" podID="9ec01088-cc90-4188-a393-e872a5f5be60" containerID="d1f56e55753ef464befa07e408f1fb8ceac722cd2affffa9a172466c77c6484c" exitCode=1
Sep 17 21:58:42.581286 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.581263    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerStarted","Data":"a8021b111fa86fe466507aa2fb4ab0890bfd19896fb6b5b4934ebc7975a2d03c"}
Sep 17 21:58:42.581356 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.581341    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerStarted","Data":"e68149fa318d39a7b23ebed20f7b77bd31b40c314dd6ce542b3371f0986e3f5e"}
Sep 17 21:58:42.581412 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.581397    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerStarted","Data":"26456824d80b949300de2ec2510fb8ff07f5249f424c1ca02659bb9df7ead9ef"}
Sep 17 21:58:42.581491 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.581470    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerStarted","Data":"38005f19469ebee5b70a563c1b8233ba8fcb99d127afd15256cc302b3cdd8c6f"}
Sep 17 21:58:42.581548 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.581524    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerDied","Data":"d1f56e55753ef464befa07e408f1fb8ceac722cd2affffa9a172466c77c6484c"}
Sep 17 21:58:42.582627 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.582609    2585 generic.go:356] "Generic (PLEG): container finished" podID="5116c2df-4d3f-4892-91a3-b04ffe887787" containerID="9524cb7f4a60457a1daaf70f6220e8eca6489cbef9723e48da2c11050bfd9c4a" exitCode=0
Sep 17 21:58:42.582718 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.582697    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcl8l" event={"ID":"5116c2df-4d3f-4892-91a3-b04ffe887787","Type":"ContainerDied","Data":"9524cb7f4a60457a1daaf70f6220e8eca6489cbef9723e48da2c11050bfd9c4a"}
Sep 17 21:58:42.583188 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.583168    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:42.583282 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.583234    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rfrx9"
Sep 17 21:58:42.592758 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.592723    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9rxzt" podStartSLOduration=4.725569286 podStartE2EDuration="22.592710919s" totalImagesPullingTime="17.867141633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:42.59222294 +0000 UTC m=+24.454476418" watchObservedRunningTime="2026-09-17 21:58:42.592710919 +0000 UTC m=+24.454964396"
Sep 17 21:58:42.732441 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:42.732403    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 21:58:43.424062 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:43.423952    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T21:58:42.732419906Z","UUID":"70efc526-2887-4759-ad38-0097bb825593","Handler":null,"Name":"","Endpoint":""}
Sep 17 21:58:43.425769 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:43.425688    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 21:58:43.425769 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:43.425724    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 21:58:43.587864 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:43.587582    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f64cd" event={"ID":"50e0dc15-2515-4770-8887-9643beec1691","Type":"ContainerStarted","Data":"98c649ccd967419cbd9306ace4e8c24562e93e0fac42ff495882f9aae7e6b781"}
Sep 17 21:58:43.590731 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:43.590695    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn" event={"ID":"3bc8002f-2d01-4adc-8b18-d657102de70a","Type":"ContainerStarted","Data":"834d40562e2eaceb7db813c757d90eabfd52a045f839f5f4adc9f7574b5ba4de"}
Sep 17 21:58:43.604282 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:43.604239    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5lmb6" podStartSLOduration=5.721894971 podStartE2EDuration="23.604226489s" totalImagesPullingTime="17.882331518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:42.642531065 +0000 UTC m=+24.504784541" watchObservedRunningTime="2026-09-17 21:58:43.604226489 +0000 UTC m=+25.466479974"
Sep 17 21:58:43.604720 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:43.604693    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f64cd" podStartSLOduration=5.737757103 podStartE2EDuration="23.604687565s" totalImagesPullingTime="17.866930462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:43.60401719 +0000 UTC m=+25.466270667" watchObservedRunningTime="2026-09-17 21:58:43.604687565 +0000 UTC m=+25.466941040"
Sep 17 21:58:44.459246 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:44.459165    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:44.459412 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:44.459166    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:44.459412 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:44.459282    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:44.459412 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:44.459390    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:44.459412 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:44.459166    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:44.459640 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:44.459496    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:44.596332 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:44.596307    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 21:58:44.596782 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:44.596755    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerStarted","Data":"9f323f89479384b75aa24a53c9b31caf118d218953b9686e39e4d208c0c78a51"}
Sep 17 21:58:44.599169 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:44.599112    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn" event={"ID":"3bc8002f-2d01-4adc-8b18-d657102de70a","Type":"ContainerStarted","Data":"1ad93a56274249c6dbbd1a76fdd920a408b049a72ca4cc05677a4022935a4b90"}
Sep 17 21:58:44.629357 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:44.629314    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbwcn" podStartSLOduration=4.705201831 podStartE2EDuration="24.629298154s" totalImagesPullingTime="19.924096323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:44.628784443 +0000 UTC m=+26.491037919" watchObservedRunningTime="2026-09-17 21:58:44.629298154 +0000 UTC m=+26.491551646"
Sep 17 21:58:46.459403 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:46.459123    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:46.459791 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:46.459123    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:46.459791 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:46.459164    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:46.459791 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:46.459496    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:46.459791 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:46.459566    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:46.459791 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:46.459610    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:47.609206 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:47.609180    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 21:58:47.610037 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:47.609622    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerStarted","Data":"0f0f5ea3d9ff3917003c1bba2de1556a26ac65ea369519c3604218d326e4df41"}
Sep 17 21:58:47.610037 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:47.609893    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:47.610151 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:47.610036    2585 scope.go:117] "RemoveContainer" containerID="d1f56e55753ef464befa07e408f1fb8ceac722cd2affffa9a172466c77c6484c"
Sep 17 21:58:47.611938 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:47.611743    2585 generic.go:356] "Generic (PLEG): container finished" podID="5116c2df-4d3f-4892-91a3-b04ffe887787" containerID="da583eb8c43be703b48c7c0ca6bdd6f6167d2aea0d0b22f271ce6022445e6c6e" exitCode=0
Sep 17 21:58:47.612036 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:47.611816    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcl8l" event={"ID":"5116c2df-4d3f-4892-91a3-b04ffe887787","Type":"ContainerDied","Data":"da583eb8c43be703b48c7c0ca6bdd6f6167d2aea0d0b22f271ce6022445e6c6e"}
Sep 17 21:58:47.628120 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:47.628101    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:48.459302 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.459270    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:48.459302 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.459286    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:48.459478 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:48.459375    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:48.459478 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:48.459417    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:48.459598 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.459487    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:48.459598 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:48.459586    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:48.551320 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.551293    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tr566"]
Sep 17 21:58:48.556310 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.556285    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2r9c"]
Sep 17 21:58:48.556959 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.556937    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lm5n6"]
Sep 17 21:58:48.618399 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.618376    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 21:58:48.618829 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.618806    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" event={"ID":"9ec01088-cc90-4188-a393-e872a5f5be60","Type":"ContainerStarted","Data":"21fde429d31a03a40daf29bb40ec7e5f9646a1df9d708ffb47034a92d2b30ccc"}
Sep 17 21:58:48.619180 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.619163    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:48.619269 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.619236    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:48.621112 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.621093    2585 generic.go:356] "Generic (PLEG): container finished" podID="5116c2df-4d3f-4892-91a3-b04ffe887787" containerID="9a1ebcec0b7a5e1c46361a8d83abab63e1ea1c40b0462513dca261137fa51c2f" exitCode=0
Sep 17 21:58:48.621194 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.621153    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:48.621233 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.621188    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcl8l" event={"ID":"5116c2df-4d3f-4892-91a3-b04ffe887787","Type":"ContainerDied","Data":"9a1ebcec0b7a5e1c46361a8d83abab63e1ea1c40b0462513dca261137fa51c2f"}
Sep 17 21:58:48.621375 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:48.621343    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:48.621407 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.621382    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:48.621493 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:48.621477    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:48.621541 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.621511    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:48.621576 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:48.621560    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:48.635545 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.635526    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:58:48.647109 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:48.647068    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd" podStartSLOduration=10.772464958 podStartE2EDuration="28.647055243s" totalImagesPullingTime="17.874590285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:48.645591957 +0000 UTC m=+30.507845446" watchObservedRunningTime="2026-09-17 21:58:48.647055243 +0000 UTC m=+30.509308718"
Sep 17 21:58:49.626351 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:49.626327    2585 generic.go:356] "Generic (PLEG): container finished" podID="5116c2df-4d3f-4892-91a3-b04ffe887787" containerID="43dab11cda3e5bf4302fcf47f9ce714d30ed73bf8f1285ec36190cdd803a9684" exitCode=0
Sep 17 21:58:49.626675 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:49.626362    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcl8l" event={"ID":"5116c2df-4d3f-4892-91a3-b04ffe887787","Type":"ContainerDied","Data":"43dab11cda3e5bf4302fcf47f9ce714d30ed73bf8f1285ec36190cdd803a9684"}
Sep 17 21:58:50.462542 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:50.462514    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:50.462695 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:50.462553    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:50.462695 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:50.462616    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:50.462813 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:50.462763    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:50.462813 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:50.462514    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:50.462937 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:50.462915    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:52.459122 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:52.459089    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:52.459584 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:52.459129    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:52.459584 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:52.459093    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:52.459584 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:52.459214    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tr566" podUID="45d76677-9235-42a6-8963-04df258d406a"
Sep 17 21:58:52.459584 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:52.459292    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m2r9c" podUID="e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb"
Sep 17 21:58:52.459584 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:52.459382    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lm5n6" podUID="12a3a05c-bf13-4011-a95e-36271b66ec5d"
Sep 17 21:58:54.066437 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.066404    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-232.ec2.internal" event="NodeReady"
Sep 17 21:58:54.066797 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.066550    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 21:58:54.087494 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.087470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:54.087632 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.087615    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:54.087688 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.087678    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret podName:45d76677-9235-42a6-8963-04df258d406a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:26.087663244 +0000 UTC m=+67.949916698 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret") pod "global-pull-secret-syncer-tr566" (UID: "45d76677-9235-42a6-8963-04df258d406a") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:54.137310 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.137270    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z5rqh"]
Sep 17 21:58:54.148239 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.148204    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zlf9g"]
Sep 17 21:58:54.148405 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.148335    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.151109 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.151086    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 21:58:54.151376 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.151357    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 21:58:54.151483 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.151466    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb92b\""
Sep 17 21:58:54.164444 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.164404    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5rqh"]
Sep 17 21:58:54.164571 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.164460    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zlf9g"]
Sep 17 21:58:54.164571 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.164554    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zlf9g"
Sep 17 21:58:54.168035 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.168016    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 21:58:54.168130 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.168067    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 21:58:54.168323 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.168305    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 21:58:54.168563 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.168550    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hxw4c\""
Sep 17 21:58:54.188163 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.188139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:54.188283 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.188239    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:54.188348 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.188307    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:54.188402 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.188371    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:54.188402 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.188378    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs podName:12a3a05c-bf13-4011-a95e-36271b66ec5d nodeName:}" failed. No retries permitted until 2026-09-17 21:59:26.188358232 +0000 UTC m=+68.050611694 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs") pod "network-metrics-daemon-lm5n6" (UID: "12a3a05c-bf13-4011-a95e-36271b66ec5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:54.188402 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.188392    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:54.188566 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.188405    2585 projected.go:196] Error preparing data for projected volume kube-api-access-f5jkh for pod openshift-network-diagnostics/network-check-target-m2r9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:54.188566 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.188494    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh podName:e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb nodeName:}" failed. No retries permitted until 2026-09-17 21:59:26.188477797 +0000 UTC m=+68.050731250 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f5jkh" (UniqueName: "kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh") pod "network-check-target-m2r9c" (UID: "e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:54.289470 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.289420    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.289613 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.289491    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-config-volume\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.289613 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.289529    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsdc\" (UniqueName: \"kubernetes.io/projected/269460ba-d873-42ce-a6c0-e6b4d64fc627-kube-api-access-tnsdc\") pod \"ingress-canary-zlf9g\" (UID: \"269460ba-d873-42ce-a6c0-e6b4d64fc627\") " pod="openshift-ingress-canary/ingress-canary-zlf9g"
Sep 17 21:58:54.289613 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.289564    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9nn\" (UniqueName: \"kubernetes.io/projected/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-kube-api-access-4l9nn\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.289753 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.289638    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-tmp-dir\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.289753 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.289673    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/269460ba-d873-42ce-a6c0-e6b4d64fc627-cert\") pod \"ingress-canary-zlf9g\" (UID: \"269460ba-d873-42ce-a6c0-e6b4d64fc627\") " pod="openshift-ingress-canary/ingress-canary-zlf9g"
Sep 17 21:58:54.390840 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.390803    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-tmp-dir\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.390840 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.390840    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/269460ba-d873-42ce-a6c0-e6b4d64fc627-cert\") pod \"ingress-canary-zlf9g\" (UID: \"269460ba-d873-42ce-a6c0-e6b4d64fc627\") " pod="openshift-ingress-canary/ingress-canary-zlf9g"
Sep 17 21:58:54.391061 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.390910    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.391061 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.390956    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-config-volume\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.391061 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.390984    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsdc\" (UniqueName: \"kubernetes.io/projected/269460ba-d873-42ce-a6c0-e6b4d64fc627-kube-api-access-tnsdc\") pod \"ingress-canary-zlf9g\" (UID: \"269460ba-d873-42ce-a6c0-e6b4d64fc627\") " pod="openshift-ingress-canary/ingress-canary-zlf9g"
Sep 17 21:58:54.391061 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.391019    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9nn\" (UniqueName: \"kubernetes.io/projected/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-kube-api-access-4l9nn\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.391061 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.391040    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:54.391279 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.391123    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls podName:a0157d6a-2e6f-4af0-88ad-842ddc9003ed nodeName:}" failed. No retries permitted until 2026-09-17 21:58:54.8911038 +0000 UTC m=+36.753357275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls") pod "dns-default-z5rqh" (UID: "a0157d6a-2e6f-4af0-88ad-842ddc9003ed") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:54.391279 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.391206    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-tmp-dir\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.391584 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.391563    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-config-volume\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.395306 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.395283    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/269460ba-d873-42ce-a6c0-e6b4d64fc627-cert\") pod \"ingress-canary-zlf9g\" (UID: \"269460ba-d873-42ce-a6c0-e6b4d64fc627\") " pod="openshift-ingress-canary/ingress-canary-zlf9g"
Sep 17 21:58:54.402695 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.402671    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9nn\" (UniqueName: \"kubernetes.io/projected/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-kube-api-access-4l9nn\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.403644 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.403624    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsdc\" (UniqueName: \"kubernetes.io/projected/269460ba-d873-42ce-a6c0-e6b4d64fc627-kube-api-access-tnsdc\") pod \"ingress-canary-zlf9g\" (UID: \"269460ba-d873-42ce-a6c0-e6b4d64fc627\") " pod="openshift-ingress-canary/ingress-canary-zlf9g"
Sep 17 21:58:54.458856 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.458777    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:58:54.458977 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.458778    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:58:54.458977 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.458795    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:58:54.461660 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.461638    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:58:54.461778 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.461712    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:58:54.461944 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.461925    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pzq2x\""
Sep 17 21:58:54.462043 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.461980    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:58:54.463762 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.462198    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:58:54.463762 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.462214    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qd56j\""
Sep 17 21:58:54.474157 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.474140    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zlf9g"
Sep 17 21:58:54.640381 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.640351    2585 generic.go:356] "Generic (PLEG): container finished" podID="5116c2df-4d3f-4892-91a3-b04ffe887787" containerID="be4005798d7aaa356e565d9ee41594cfaea9caf779d6b9e7f6147d0cc5732fd1" exitCode=0
Sep 17 21:58:54.640519 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.640406    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcl8l" event={"ID":"5116c2df-4d3f-4892-91a3-b04ffe887787","Type":"ContainerDied","Data":"be4005798d7aaa356e565d9ee41594cfaea9caf779d6b9e7f6147d0cc5732fd1"}
Sep 17 21:58:54.790477 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.790453    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zlf9g"]
Sep 17 21:58:54.793159 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:58:54.793135    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269460ba_d873_42ce_a6c0_e6b4d64fc627.slice/crio-36f17ac00cdc3670ee4cb7e8384836fd09a36dffac6fb9d32b8c7fc6ba1d8a1e WatchSource:0}: Error finding container 36f17ac00cdc3670ee4cb7e8384836fd09a36dffac6fb9d32b8c7fc6ba1d8a1e: Status 404 returned error can't find the container with id 36f17ac00cdc3670ee4cb7e8384836fd09a36dffac6fb9d32b8c7fc6ba1d8a1e
Sep 17 21:58:54.893848 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:54.893819    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:54.893963 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.893938    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:54.894001 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:54.893994    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls podName:a0157d6a-2e6f-4af0-88ad-842ddc9003ed nodeName:}" failed. No retries permitted until 2026-09-17 21:58:55.893977957 +0000 UTC m=+37.756231415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls") pod "dns-default-z5rqh" (UID: "a0157d6a-2e6f-4af0-88ad-842ddc9003ed") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:55.644699 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:55.644460    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zlf9g" event={"ID":"269460ba-d873-42ce-a6c0-e6b4d64fc627","Type":"ContainerStarted","Data":"36f17ac00cdc3670ee4cb7e8384836fd09a36dffac6fb9d32b8c7fc6ba1d8a1e"}
Sep 17 21:58:55.648312 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:55.648267    2585 generic.go:356] "Generic (PLEG): container finished" podID="5116c2df-4d3f-4892-91a3-b04ffe887787" containerID="f013439b66d163c168de5a2a195cf4d49ddd871c9e9fa87545ee101c78f0462b" exitCode=0
Sep 17 21:58:55.648312 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:55.648311    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcl8l" event={"ID":"5116c2df-4d3f-4892-91a3-b04ffe887787","Type":"ContainerDied","Data":"f013439b66d163c168de5a2a195cf4d49ddd871c9e9fa87545ee101c78f0462b"}
Sep 17 21:58:55.901624 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:55.901524    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:55.901749 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:55.901672    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:55.901749 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:55.901731    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls podName:a0157d6a-2e6f-4af0-88ad-842ddc9003ed nodeName:}" failed. No retries permitted until 2026-09-17 21:58:57.901714701 +0000 UTC m=+39.763968178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls") pod "dns-default-z5rqh" (UID: "a0157d6a-2e6f-4af0-88ad-842ddc9003ed") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:56.654680 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:56.654644    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qcl8l" event={"ID":"5116c2df-4d3f-4892-91a3-b04ffe887787","Type":"ContainerStarted","Data":"46169e40511788e989b797201253665b8afc656a558901e31e5dc5f96693c03f"}
Sep 17 21:58:56.677163 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:56.677077    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qcl8l" podStartSLOduration=9.162887081 podStartE2EDuration="36.67705412s" totalImagesPullingTime="27.514167039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:56.676832729 +0000 UTC m=+38.539086205" watchObservedRunningTime="2026-09-17 21:58:56.67705412 +0000 UTC m=+38.539307617"
Sep 17 21:58:57.658450 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:57.658399    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zlf9g" event={"ID":"269460ba-d873-42ce-a6c0-e6b4d64fc627","Type":"ContainerStarted","Data":"4564b27a2adfec7df04eab8967c266d2a94c35f1ee71d7cba191c56d36977960"}
Sep 17 21:58:57.918403 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:58:57.918336    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:58:57.918539 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:57.918482    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:57.918581 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:58:57.918546    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls podName:a0157d6a-2e6f-4af0-88ad-842ddc9003ed nodeName:}" failed. No retries permitted until 2026-09-17 21:59:01.918527325 +0000 UTC m=+43.780780781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls") pod "dns-default-z5rqh" (UID: "a0157d6a-2e6f-4af0-88ad-842ddc9003ed") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:01.950413 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:01.950376    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:59:01.950797 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:01.950501    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:01.950797 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:01.950550    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls podName:a0157d6a-2e6f-4af0-88ad-842ddc9003ed nodeName:}" failed. No retries permitted until 2026-09-17 21:59:09.950535808 +0000 UTC m=+51.812789262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls") pod "dns-default-z5rqh" (UID: "a0157d6a-2e6f-4af0-88ad-842ddc9003ed") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:10.004186 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:10.004149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:59:10.004555 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:10.004262    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:10.004555 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:10.004330    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls podName:a0157d6a-2e6f-4af0-88ad-842ddc9003ed nodeName:}" failed. No retries permitted until 2026-09-17 21:59:26.004317209 +0000 UTC m=+67.866570663 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls") pod "dns-default-z5rqh" (UID: "a0157d6a-2e6f-4af0-88ad-842ddc9003ed") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:16.324248 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.324193    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zlf9g" podStartSLOduration=19.98410569 podStartE2EDuration="22.324176526s" totalImagesPullingTime="2.340070836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:57.674846931 +0000 UTC m=+39.537100408" watchObservedRunningTime="2026-09-17 21:59:16.324176526 +0000 UTC m=+58.186430007"
Sep 17 21:59:16.324724 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.324484    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6"]
Sep 17 21:59:16.327328 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.327310    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6"
Sep 17 21:59:16.329714 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.329690    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:16.329714 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.329705    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:16.330836 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.330815    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mntv4\""
Sep 17 21:59:16.331833 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.331810    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"]
Sep 17 21:59:16.334219 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.334193    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"]
Sep 17 21:59:16.334898 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.334874    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:16.336991 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.336966    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl"]
Sep 17 21:59:16.337371 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.337321    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.338675 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.338651    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 21:59:16.338776 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.338675    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:16.338850 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.338778    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:16.338980 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.338965    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p4kpn\""
Sep 17 21:59:16.339668 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.339654    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 21:59:16.339869 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.339854    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 21:59:16.340092 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.340076    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cvckd\""
Sep 17 21:59:16.340171 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.340122    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl"
Sep 17 21:59:16.340394 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.340380    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:16.340493 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.340464    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:16.341883 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.341865    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6"]
Sep 17 21:59:16.342767 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.342749    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hz779\""
Sep 17 21:59:16.349823 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.349803    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"]
Sep 17 21:59:16.350724 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.350705    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"]
Sep 17 21:59:16.360538 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.360519    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl"]
Sep 17 21:59:16.430641 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.430600    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"]
Sep 17 21:59:16.432309 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.432293    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:16.434365 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.434345    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 21:59:16.434462 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.434451    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 21:59:16.434765 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.434750    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 21:59:16.434867 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.434852    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xdgfh\""
Sep 17 21:59:16.435725 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.435709    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 21:59:16.447342 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.447319    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnrm\" (UniqueName: \"kubernetes.io/projected/7b514b24-b573-493e-bd63-ff52b7f4a787-kube-api-access-2dnrm\") pod \"kube-storage-version-migrator-operator-55889f7b47-mh7vm\" (UID: \"7b514b24-b573-493e-bd63-ff52b7f4a787\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.447477 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.447354    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7sn\" (UniqueName: \"kubernetes.io/projected/b873238b-edf6-4f37-844f-762f7d13b0a0-kube-api-access-tc7sn\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:16.447477 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.447374    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b514b24-b573-493e-bd63-ff52b7f4a787-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-mh7vm\" (UID: \"7b514b24-b573-493e-bd63-ff52b7f4a787\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.447477 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.447388    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zf5\" (UniqueName: \"kubernetes.io/projected/af9f97fa-dccd-4256-bd7d-a1ac2d005eb8-kube-api-access-x4zf5\") pod \"network-check-source-7cbc45fddc-jqwrl\" (UID: \"af9f97fa-dccd-4256-bd7d-a1ac2d005eb8\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl"
Sep 17 21:59:16.447586 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.447510    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v4gf\" (UniqueName: \"kubernetes.io/projected/bdf61d98-cf89-4bae-b903-48689855cba0-kube-api-access-4v4gf\") pod \"volume-data-source-validator-5c4db8bdf9-5rcm6\" (UID: \"bdf61d98-cf89-4bae-b903-48689855cba0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6"
Sep 17 21:59:16.447586 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.447545    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:16.447586 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.447576    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b514b24-b573-493e-bd63-ff52b7f4a787-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-mh7vm\" (UID: \"7b514b24-b573-493e-bd63-ff52b7f4a787\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.450346 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.450326    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"]
Sep 17 21:59:16.548664 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.548629    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4gf\" (UniqueName: \"kubernetes.io/projected/bdf61d98-cf89-4bae-b903-48689855cba0-kube-api-access-4v4gf\") pod \"volume-data-source-validator-5c4db8bdf9-5rcm6\" (UID: \"bdf61d98-cf89-4bae-b903-48689855cba0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6"
Sep 17 21:59:16.548804 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.548684    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:16.548804 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.548711    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b514b24-b573-493e-bd63-ff52b7f4a787-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-mh7vm\" (UID: \"7b514b24-b573-493e-bd63-ff52b7f4a787\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.548804 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.548740    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnrm\" (UniqueName: \"kubernetes.io/projected/7b514b24-b573-493e-bd63-ff52b7f4a787-kube-api-access-2dnrm\") pod \"kube-storage-version-migrator-operator-55889f7b47-mh7vm\" (UID: \"7b514b24-b573-493e-bd63-ff52b7f4a787\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.548804 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.548770    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gqpg\" (UniqueName: \"kubernetes.io/projected/43882037-9994-4040-a7cd-bc334661ace1-kube-api-access-5gqpg\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:16.548973 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.548913    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/43882037-9994-4040-a7cd-bc334661ace1-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:16.548973 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:16.548947    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:16.549047 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.548949    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7sn\" (UniqueName: \"kubernetes.io/projected/b873238b-edf6-4f37-844f-762f7d13b0a0-kube-api-access-tc7sn\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:16.549047 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:16.549024    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls podName:b873238b-edf6-4f37-844f-762f7d13b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:17.04900503 +0000 UTC m=+58.911258489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8fnn2" (UID: "b873238b-edf6-4f37-844f-762f7d13b0a0") : secret "samples-operator-tls" not found
Sep 17 21:59:16.549163 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.549065    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:16.549163 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.549098    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b514b24-b573-493e-bd63-ff52b7f4a787-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-mh7vm\" (UID: \"7b514b24-b573-493e-bd63-ff52b7f4a787\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.549163 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.549125    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zf5\" (UniqueName: \"kubernetes.io/projected/af9f97fa-dccd-4256-bd7d-a1ac2d005eb8-kube-api-access-x4zf5\") pod \"network-check-source-7cbc45fddc-jqwrl\" (UID: \"af9f97fa-dccd-4256-bd7d-a1ac2d005eb8\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl"
Sep 17 21:59:16.550136 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.550118    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b514b24-b573-493e-bd63-ff52b7f4a787-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-mh7vm\" (UID: \"7b514b24-b573-493e-bd63-ff52b7f4a787\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.552300 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.552282    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b514b24-b573-493e-bd63-ff52b7f4a787-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-mh7vm\" (UID: \"7b514b24-b573-493e-bd63-ff52b7f4a787\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.559319 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.559294    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnrm\" (UniqueName: \"kubernetes.io/projected/7b514b24-b573-493e-bd63-ff52b7f4a787-kube-api-access-2dnrm\") pod \"kube-storage-version-migrator-operator-55889f7b47-mh7vm\" (UID: \"7b514b24-b573-493e-bd63-ff52b7f4a787\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.559733 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.559716    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4gf\" (UniqueName: \"kubernetes.io/projected/bdf61d98-cf89-4bae-b903-48689855cba0-kube-api-access-4v4gf\") pod \"volume-data-source-validator-5c4db8bdf9-5rcm6\" (UID: \"bdf61d98-cf89-4bae-b903-48689855cba0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6"
Sep 17 21:59:16.559782 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.559756    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zf5\" (UniqueName: \"kubernetes.io/projected/af9f97fa-dccd-4256-bd7d-a1ac2d005eb8-kube-api-access-x4zf5\") pod \"network-check-source-7cbc45fddc-jqwrl\" (UID: \"af9f97fa-dccd-4256-bd7d-a1ac2d005eb8\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl"
Sep 17 21:59:16.560245 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.560225    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7sn\" (UniqueName: \"kubernetes.io/projected/b873238b-edf6-4f37-844f-762f7d13b0a0-kube-api-access-tc7sn\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:16.639090 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.639042    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6"
Sep 17 21:59:16.649888 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.649865    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gqpg\" (UniqueName: \"kubernetes.io/projected/43882037-9994-4040-a7cd-bc334661ace1-kube-api-access-5gqpg\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:16.649948 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.649916    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/43882037-9994-4040-a7cd-bc334661ace1-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:16.650118 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.650089    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:16.650203 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:16.650177    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:16.650251 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:16.650241    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls podName:43882037-9994-4040-a7cd-bc334661ace1 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:17.150223754 +0000 UTC m=+59.012477211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdd9f" (UID: "43882037-9994-4040-a7cd-bc334661ace1") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:16.650712 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.650692    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/43882037-9994-4040-a7cd-bc334661ace1-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:16.655008 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.654985    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"
Sep 17 21:59:16.659724 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.659704    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl"
Sep 17 21:59:16.661988 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.661970    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gqpg\" (UniqueName: \"kubernetes.io/projected/43882037-9994-4040-a7cd-bc334661ace1-kube-api-access-5gqpg\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:16.910061 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.909978    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6"]
Sep 17 21:59:16.913446 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:16.913395    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf61d98_cf89_4bae_b903_48689855cba0.slice/crio-865d252337e2f3184b5256b21b6b7067b514e5d2d097e38da5b20b349b852e9e WatchSource:0}: Error finding container 865d252337e2f3184b5256b21b6b7067b514e5d2d097e38da5b20b349b852e9e: Status 404 returned error can't find the container with id 865d252337e2f3184b5256b21b6b7067b514e5d2d097e38da5b20b349b852e9e
Sep 17 21:59:16.934529 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.934510    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl"]
Sep 17 21:59:16.937218 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:16.937196    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm"]
Sep 17 21:59:16.937498 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:16.937474    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9f97fa_dccd_4256_bd7d_a1ac2d005eb8.slice/crio-ef10e95bacb4bc5417ddbda6f369c8642d119ae7000f44684158b6c1389b5aa7 WatchSource:0}: Error finding container ef10e95bacb4bc5417ddbda6f369c8642d119ae7000f44684158b6c1389b5aa7: Status 404 returned error can't find the container with id ef10e95bacb4bc5417ddbda6f369c8642d119ae7000f44684158b6c1389b5aa7
Sep 17 21:59:16.939854 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:16.939834    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b514b24_b573_493e_bd63_ff52b7f4a787.slice/crio-67cbd10c16ee2e5f0d89760ce8e85780ead6285310068a8b7f2847d959f40e2d WatchSource:0}: Error finding container 67cbd10c16ee2e5f0d89760ce8e85780ead6285310068a8b7f2847d959f40e2d: Status 404 returned error can't find the container with id 67cbd10c16ee2e5f0d89760ce8e85780ead6285310068a8b7f2847d959f40e2d
Sep 17 21:59:17.052909 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:17.052886    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:17.053009 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:17.052987    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:17.053049 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:17.053032    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls podName:b873238b-edf6-4f37-844f-762f7d13b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:18.053020173 +0000 UTC m=+59.915273627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8fnn2" (UID: "b873238b-edf6-4f37-844f-762f7d13b0a0") : secret "samples-operator-tls" not found
Sep 17 21:59:17.153591 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:17.153566    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:17.153707 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:17.153693    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:17.153758 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:17.153746    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls podName:43882037-9994-4040-a7cd-bc334661ace1 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:18.153730696 +0000 UTC m=+60.015984150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdd9f" (UID: "43882037-9994-4040-a7cd-bc334661ace1") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:17.716340 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:17.716242    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm" event={"ID":"7b514b24-b573-493e-bd63-ff52b7f4a787","Type":"ContainerStarted","Data":"67cbd10c16ee2e5f0d89760ce8e85780ead6285310068a8b7f2847d959f40e2d"}
Sep 17 21:59:17.718202 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:17.718167    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6" event={"ID":"bdf61d98-cf89-4bae-b903-48689855cba0","Type":"ContainerStarted","Data":"865d252337e2f3184b5256b21b6b7067b514e5d2d097e38da5b20b349b852e9e"}
Sep 17 21:59:17.720183 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:17.720139    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl" event={"ID":"af9f97fa-dccd-4256-bd7d-a1ac2d005eb8","Type":"ContainerStarted","Data":"ef10e95bacb4bc5417ddbda6f369c8642d119ae7000f44684158b6c1389b5aa7"}
Sep 17 21:59:18.059613 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:18.059579    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:18.059911 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:18.059706    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:18.059911 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:18.059775    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls podName:b873238b-edf6-4f37-844f-762f7d13b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.059760295 +0000 UTC m=+61.922013754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8fnn2" (UID: "b873238b-edf6-4f37-844f-762f7d13b0a0") : secret "samples-operator-tls" not found
Sep 17 21:59:18.160192 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:18.160153    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:18.160481 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:18.160330    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:18.160481 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:18.160411    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls podName:43882037-9994-4040-a7cd-bc334661ace1 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.160388552 +0000 UTC m=+62.022642013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdd9f" (UID: "43882037-9994-4040-a7cd-bc334661ace1") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:18.725457 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:18.725101    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6" event={"ID":"bdf61d98-cf89-4bae-b903-48689855cba0","Type":"ContainerStarted","Data":"da9a81b8752954e52e722987fca41739ef224fff142bed2683cca3b17b62a501"}
Sep 17 21:59:18.740944 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:18.740898    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-5rcm6" podStartSLOduration=1.706896389 podStartE2EDuration="2.740883365s" totalImagesPullingTime="1.033986976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:18.739005338 +0000 UTC m=+60.601258814" watchObservedRunningTime="2026-09-17 21:59:18.740883365 +0000 UTC m=+60.603136840"
Sep 17 21:59:20.072694 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.072657    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:20.073120 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:20.072814    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:20.073120 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:20.072889    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls podName:b873238b-edf6-4f37-844f-762f7d13b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.072867928 +0000 UTC m=+65.935121382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8fnn2" (UID: "b873238b-edf6-4f37-844f-762f7d13b0a0") : secret "samples-operator-tls" not found
Sep 17 21:59:20.173249 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.173178    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:20.173392 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:20.173319    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:20.173392 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:20.173384    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls podName:43882037-9994-4040-a7cd-bc334661ace1 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.173368654 +0000 UTC m=+66.035622111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdd9f" (UID: "43882037-9994-4040-a7cd-bc334661ace1") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:20.638877 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.638850    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9ddd"
Sep 17 21:59:20.732448 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.732390    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl" event={"ID":"af9f97fa-dccd-4256-bd7d-a1ac2d005eb8","Type":"ContainerStarted","Data":"8a4db39b307636c3d6a5e913c8cec95d56a61dccd7e99f2e53fdfe100dae3aae"}
Sep 17 21:59:20.734120 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.734091    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm" event={"ID":"7b514b24-b573-493e-bd63-ff52b7f4a787","Type":"ContainerStarted","Data":"52587c7159d1b5382ea71a7bd629b020bd824703fbb924eb2283a948d4ee9c88"}
Sep 17 21:59:20.800785 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.800734    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm" podStartSLOduration=1.825137093 podStartE2EDuration="4.800715116s" totalImagesPullingTime="2.975578023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:20.800243455 +0000 UTC m=+62.662496932" watchObservedRunningTime="2026-09-17 21:59:20.800715116 +0000 UTC m=+62.662968593"
Sep 17 21:59:20.801258 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.801232    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jqwrl" podStartSLOduration=1.7985744989999999 podStartE2EDuration="4.801223333s" totalImagesPullingTime="3.002648834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:20.759741728 +0000 UTC m=+62.621995204" watchObservedRunningTime="2026-09-17 21:59:20.801223333 +0000 UTC m=+62.663476809"
Sep 17 21:59:20.842206 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.842172    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h8ccm"]
Sep 17 21:59:20.845823 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.845800    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.848294 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.848135    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wm66n\""
Sep 17 21:59:20.848294 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.848162    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 21:59:20.848294 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.848178    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 21:59:20.848294 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.848244    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 21:59:20.848574 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.848371    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 21:59:20.860144 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.859976    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h8ccm"]
Sep 17 21:59:20.880297 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.880271    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c540354a-ac88-46ed-8a86-bec4fbb8c082-crio-socket\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.880389 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.880331    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c540354a-ac88-46ed-8a86-bec4fbb8c082-data-volume\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.880389 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.880355    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.880389 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.880380    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkqb4\" (UniqueName: \"kubernetes.io/projected/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-api-access-qkqb4\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.880548 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.880396    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.981240 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.981159    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c540354a-ac88-46ed-8a86-bec4fbb8c082-data-volume\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.981240 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.981203    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.981462 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.981244    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkqb4\" (UniqueName: \"kubernetes.io/projected/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-api-access-qkqb4\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.981462 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.981271    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.981462 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.981322    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c540354a-ac88-46ed-8a86-bec4fbb8c082-crio-socket\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.981462 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:20.981362    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.981462 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:20.981398    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:20.981462 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.981405    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c540354a-ac88-46ed-8a86-bec4fbb8c082-crio-socket\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.981700 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:20.981472    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.481422449 +0000 UTC m=+63.343675919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.981700 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:20.981500    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.481489739 +0000 UTC m=+63.343743193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:20.981700 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.981567    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c540354a-ac88-46ed-8a86-bec4fbb8c082-data-volume\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:20.992195 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:20.992163    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkqb4\" (UniqueName: \"kubernetes.io/projected/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-api-access-qkqb4\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:21.485757 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:21.485682    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:21.486191 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:21.485768    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:21.486191 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:21.485825    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:21.486191 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:21.485850    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.485830127 +0000 UTC m=+64.348083601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:21.486191 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:21.485946    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:21.486191 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:21.486020    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.485993606 +0000 UTC m=+64.348247074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.494708 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:22.494676    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:22.495212 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:22.494723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:22.495212 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:22.494780    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.495212 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:22.494845    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.494832129 +0000 UTC m=+66.357085583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.495212 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:22.494920    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.495212 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:22.494985    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.494966977 +0000 UTC m=+66.357220441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.779369 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:22.779341    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9rxzt_cdf9f479-ec36-4720-b83f-a0484c14cda4/dns-node-resolver/0.log"
Sep 17 21:59:23.580586 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:23.580560    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jbrr7_ac293f58-ae69-4c22-8804-44545908c47c/node-ca/0.log"
Sep 17 21:59:24.105630 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:24.105601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:24.105778 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:24.105706    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 21:59:24.105778 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:24.105758    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls podName:b873238b-edf6-4f37-844f-762f7d13b0a0 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:32.105745889 +0000 UTC m=+73.967999343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8fnn2" (UID: "b873238b-edf6-4f37-844f-762f7d13b0a0") : secret "samples-operator-tls" not found
Sep 17 21:59:24.206620 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:24.206579    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:24.206768 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:24.206727    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:24.206817 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:24.206783    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls podName:43882037-9994-4040-a7cd-bc334661ace1 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:32.206768716 +0000 UTC m=+74.069022171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdd9f" (UID: "43882037-9994-4040-a7cd-bc334661ace1") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:24.509248 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:24.509170    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:24.509248 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:24.509212    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:24.509410 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:24.509296    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:24.509410 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:24.509325    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:24.509410 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:24.509359    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.509342012 +0000 UTC m=+70.371595487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:24.509410 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:24.509373    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.509367334 +0000 UTC m=+70.371620788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:24.580676 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:24.580650    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zlf9g_269460ba-d873-42ce-a6c0-e6b4d64fc627/serve-healthcheck-canary/0.log"
Sep 17 21:59:26.020669 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.020632    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:59:26.021059 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:26.020744    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:26.021059 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:26.020792    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls podName:a0157d6a-2e6f-4af0-88ad-842ddc9003ed nodeName:}" failed. No retries permitted until 2026-09-17 21:59:58.020780325 +0000 UTC m=+99.883033779 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls") pod "dns-default-z5rqh" (UID: "a0157d6a-2e6f-4af0-88ad-842ddc9003ed") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:26.121658 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.121628    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:59:26.127119 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.127102    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:59:26.134790 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.134767    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/45d76677-9235-42a6-8963-04df258d406a-original-pull-secret\") pod \"global-pull-secret-syncer-tr566\" (UID: \"45d76677-9235-42a6-8963-04df258d406a\") " pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:59:26.222676 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.222650    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:59:26.222786 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.222695    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 21:59:26.225174 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.225147    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jkh\" (UniqueName: \"kubernetes.io/projected/e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb-kube-api-access-f5jkh\") pod \"network-check-target-m2r9c\" (UID: \"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb\") " pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:59:26.225719 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.225705    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:59:26.233060 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:26.233045    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 21:59:26.233137 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:26.233095    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs podName:12a3a05c-bf13-4011-a95e-36271b66ec5d nodeName:}" failed. No retries permitted until 2026-09-17 22:00:30.23308188 +0000 UTC m=+132.095335335 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs") pod "network-metrics-daemon-lm5n6" (UID: "12a3a05c-bf13-4011-a95e-36271b66ec5d") : secret "metrics-daemon-secret" not found
Sep 17 21:59:26.276529 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.276479    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tr566"
Sep 17 21:59:26.283364 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.283348    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qd56j\""
Sep 17 21:59:26.291645 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.291628    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:59:26.583532 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.583452    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tr566"]
Sep 17 21:59:26.586805 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:26.586780    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d76677_9235_42a6_8963_04df258d406a.slice/crio-110a0be7fa2c4b90fbf0a87f3061a3378c597059c3576b15b21a5e3ef124a919 WatchSource:0}: Error finding container 110a0be7fa2c4b90fbf0a87f3061a3378c597059c3576b15b21a5e3ef124a919: Status 404 returned error can't find the container with id 110a0be7fa2c4b90fbf0a87f3061a3378c597059c3576b15b21a5e3ef124a919
Sep 17 21:59:26.606097 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.606074    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m2r9c"]
Sep 17 21:59:26.608546 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:26.608526    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3dbb9bd_4956_4a4f_baf5_0fdfe8ec4dfb.slice/crio-b21b5c211204bfa7eab9c8d8e02552c771516e7a60281bacfd80020b346e8513 WatchSource:0}: Error finding container b21b5c211204bfa7eab9c8d8e02552c771516e7a60281bacfd80020b346e8513: Status 404 returned error can't find the container with id b21b5c211204bfa7eab9c8d8e02552c771516e7a60281bacfd80020b346e8513
Sep 17 21:59:26.750620 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.750591    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tr566" event={"ID":"45d76677-9235-42a6-8963-04df258d406a","Type":"ContainerStarted","Data":"110a0be7fa2c4b90fbf0a87f3061a3378c597059c3576b15b21a5e3ef124a919"}
Sep 17 21:59:26.752092 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.752070    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2r9c" event={"ID":"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb","Type":"ContainerStarted","Data":"2cc2fc8fdf235586c74f26348a3ce6873b597fab19cbff7d73b87753084028dc"}
Sep 17 21:59:26.752186 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.752112    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m2r9c" event={"ID":"e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb","Type":"ContainerStarted","Data":"b21b5c211204bfa7eab9c8d8e02552c771516e7a60281bacfd80020b346e8513"}
Sep 17 21:59:26.752230 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.752207    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:59:26.773570 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:26.773526    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m2r9c" podStartSLOduration=66.773512827 podStartE2EDuration="1m6.773512827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:26.77218812 +0000 UTC m=+68.634441607" watchObservedRunningTime="2026-09-17 21:59:26.773512827 +0000 UTC m=+68.635766332"
Sep 17 21:59:28.542413 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:28.542373    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:28.542850 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:28.542464    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:28.542850 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:28.542506    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:28.542850 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:28.542587    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:36.542566582 +0000 UTC m=+78.404820051 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:28.542850 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:28.542618    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:28.542850 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:28.542681    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls podName:c540354a-ac88-46ed-8a86-bec4fbb8c082 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:36.542666315 +0000 UTC m=+78.404919771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h8ccm" (UID: "c540354a-ac88-46ed-8a86-bec4fbb8c082") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:30.766843 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:30.766684    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tr566" event={"ID":"45d76677-9235-42a6-8963-04df258d406a","Type":"ContainerStarted","Data":"ae1319cb6fbf1d32a77ad8b3ea9c7cd14c10ea2911ef51c347bf2aabde7730a8"}
Sep 17 21:59:30.791997 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:30.791952    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tr566" podStartSLOduration=67.268744404 podStartE2EDuration="1m10.791938478s" totalImagesPullingTime="3.523194074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:30.791420596 +0000 UTC m=+72.653674073" watchObservedRunningTime="2026-09-17 21:59:30.791938478 +0000 UTC m=+72.654191954"
Sep 17 21:59:32.172751 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:32.172719    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:32.175225 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:32.175199    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b873238b-edf6-4f37-844f-762f7d13b0a0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8fnn2\" (UID: \"b873238b-edf6-4f37-844f-762f7d13b0a0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:32.251662 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:32.251640    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p4kpn\""
Sep 17 21:59:32.259137 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:32.259119    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"
Sep 17 21:59:32.273473 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:32.273420    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:32.273572 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:32.273520    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:32.273613 ip-10-0-137-232 kubenswrapper[2585]: E0917 21:59:32.273579    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls podName:43882037-9994-4040-a7cd-bc334661ace1 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:48.273562721 +0000 UTC m=+90.135816175 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdd9f" (UID: "43882037-9994-4040-a7cd-bc334661ace1") : secret "cluster-monitoring-operator-tls" not found
Sep 17 21:59:32.528122 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:32.528061    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2"]
Sep 17 21:59:32.774026 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:32.773993    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2" event={"ID":"b873238b-edf6-4f37-844f-762f7d13b0a0","Type":"ContainerStarted","Data":"fc638b65fdb1bc7b4dccbf16ce7fab3c507fe37cbf8c1c4300146b625fb352b1"}
Sep 17 21:59:34.782388 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:34.782349    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2" event={"ID":"b873238b-edf6-4f37-844f-762f7d13b0a0","Type":"ContainerStarted","Data":"cf23eeb5d98ac53c9c3ddfb5b3b68f04690155dbfe9e5fd5a6564c82366c03d2"}
Sep 17 21:59:34.782787 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:34.782410    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2" event={"ID":"b873238b-edf6-4f37-844f-762f7d13b0a0","Type":"ContainerStarted","Data":"a69b7ef2e5c7f8cb31f885e608dfc77e9cb9e6f3f5c7f62fddba6ceaf9c5f4a4"}
Sep 17 21:59:34.802681 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:34.802643    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8fnn2" podStartSLOduration=17.264426226 podStartE2EDuration="18.802630333s" totalImagesPullingTime="1.538204107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:34.802446502 +0000 UTC m=+76.664699980" watchObservedRunningTime="2026-09-17 21:59:34.802630333 +0000 UTC m=+76.664883809"
Sep 17 21:59:36.606264 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:36.606232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:36.606666 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:36.606303    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:36.606789 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:36.606771    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c540354a-ac88-46ed-8a86-bec4fbb8c082-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:36.608725 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:36.608703    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c540354a-ac88-46ed-8a86-bec4fbb8c082-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h8ccm\" (UID: \"c540354a-ac88-46ed-8a86-bec4fbb8c082\") " pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:36.757244 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:36.757217    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h8ccm"
Sep 17 21:59:37.027618 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:37.027591    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h8ccm"]
Sep 17 21:59:37.031058 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:37.031015    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc540354a_ac88_46ed_8a86_bec4fbb8c082.slice/crio-319ccc7d2509c0211c4f61f27632df46d5a6027b0a65a3e27f93d0550e3bc843 WatchSource:0}: Error finding container 319ccc7d2509c0211c4f61f27632df46d5a6027b0a65a3e27f93d0550e3bc843: Status 404 returned error can't find the container with id 319ccc7d2509c0211c4f61f27632df46d5a6027b0a65a3e27f93d0550e3bc843
Sep 17 21:59:37.792069 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:37.791978    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h8ccm" event={"ID":"c540354a-ac88-46ed-8a86-bec4fbb8c082","Type":"ContainerStarted","Data":"11a77869dacf74d992157d330bc5cd2b7811dc077f0e970bf423ba54ae072950"}
Sep 17 21:59:37.792069 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:37.792031    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h8ccm" event={"ID":"c540354a-ac88-46ed-8a86-bec4fbb8c082","Type":"ContainerStarted","Data":"319ccc7d2509c0211c4f61f27632df46d5a6027b0a65a3e27f93d0550e3bc843"}
Sep 17 21:59:38.797898 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:38.797862    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h8ccm" event={"ID":"c540354a-ac88-46ed-8a86-bec4fbb8c082","Type":"ContainerStarted","Data":"e1b5bf028b5e1298bf5404e5dadb7dcc018c0ed233acb2bbab931fc22ba96110"}
Sep 17 21:59:39.802871 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:39.802826    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h8ccm" event={"ID":"c540354a-ac88-46ed-8a86-bec4fbb8c082","Type":"ContainerStarted","Data":"6466598079b0ad8f1f62e74e29e9daf3ac983042a1c35f95aed80424ef33142c"}
Sep 17 21:59:39.822229 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:39.822182    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h8ccm" podStartSLOduration=17.9135898 podStartE2EDuration="19.822167453s" totalImagesPullingTime="1.908577653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:39.820809264 +0000 UTC m=+81.683062740" watchObservedRunningTime="2026-09-17 21:59:39.822167453 +0000 UTC m=+81.684420928"
Sep 17 21:59:45.888752 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.888719    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"]
Sep 17 21:59:45.893236 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.893215    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:45.898007 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.897978    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 21:59:45.898113 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.897999    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 21:59:45.898459 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.898420    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 21:59:45.898459 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.898452    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 21:59:45.898657 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.898457    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 21:59:45.898657 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.898443    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 21:59:45.900599 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.900581    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 21:59:45.932214 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.932193    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"]
Sep 17 21:59:45.971128 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.971099    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-ca\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:45.971244 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.971131    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:45.971244 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.971158    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:45.971244 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.971210    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-hub\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:45.971371 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.971283    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx759\" (UniqueName: \"kubernetes.io/projected/8665ab3e-e814-4d38-a138-2f8c9a4a854e-kube-api-access-qx759\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:45.971371 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.971323    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8665ab3e-e814-4d38-a138-2f8c9a4a854e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:45.998799 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:45.998773    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54f647d6c4-6hdhd"]
Sep 17 21:59:46.001625 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.001612    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.003977 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.003957    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 21:59:46.004080 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.003959    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 21:59:46.004080 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.003959    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hqrds\""
Sep 17 21:59:46.004080 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.004049    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 21:59:46.012156 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.012139    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 21:59:46.015501 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.015481    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54f647d6c4-6hdhd"]
Sep 17 21:59:46.072135 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072116    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-hub\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.072243 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072146    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58690a93-436a-45f3-ab67-16d575d3d6b9-installation-pull-secrets\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.072243 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072166    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx759\" (UniqueName: \"kubernetes.io/projected/8665ab3e-e814-4d38-a138-2f8c9a4a854e-kube-api-access-qx759\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.072243 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072185    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8665ab3e-e814-4d38-a138-2f8c9a4a854e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.072243 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072200    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58690a93-436a-45f3-ab67-16d575d3d6b9-registry-tls\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.072539 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072263    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58690a93-436a-45f3-ab67-16d575d3d6b9-bound-sa-token\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.072539 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072300    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58690a93-436a-45f3-ab67-16d575d3d6b9-image-registry-private-configuration\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.072539 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072326    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.072539 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072400    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xzq\" (UniqueName: \"kubernetes.io/projected/58690a93-436a-45f3-ab67-16d575d3d6b9-kube-api-access-98xzq\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.072539 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072456    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58690a93-436a-45f3-ab67-16d575d3d6b9-registry-certificates\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.072539 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58690a93-436a-45f3-ab67-16d575d3d6b9-ca-trust-extracted\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.072843 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072557    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-ca\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.072843 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072588    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.072843 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.072614    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58690a93-436a-45f3-ab67-16d575d3d6b9-trusted-ca\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.073047 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.073028    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8665ab3e-e814-4d38-a138-2f8c9a4a854e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.074952 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.074931    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-hub\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.075042 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.074965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-ca\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.075339 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.075320    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.075505 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.075489    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8665ab3e-e814-4d38-a138-2f8c9a4a854e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.094546 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.094522    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx759\" (UniqueName: \"kubernetes.io/projected/8665ab3e-e814-4d38-a138-2f8c9a4a854e-kube-api-access-qx759\") pod \"cluster-proxy-proxy-agent-74dc4f586c-p4bz2\" (UID: \"8665ab3e-e814-4d38-a138-2f8c9a4a854e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.173824 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.173754    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xzq\" (UniqueName: \"kubernetes.io/projected/58690a93-436a-45f3-ab67-16d575d3d6b9-kube-api-access-98xzq\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.173824 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.173788    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58690a93-436a-45f3-ab67-16d575d3d6b9-registry-certificates\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.173824 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.173807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58690a93-436a-45f3-ab67-16d575d3d6b9-ca-trust-extracted\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.174000 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.173840    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58690a93-436a-45f3-ab67-16d575d3d6b9-trusted-ca\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.174000 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.173870    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58690a93-436a-45f3-ab67-16d575d3d6b9-installation-pull-secrets\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.174000 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.173901    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58690a93-436a-45f3-ab67-16d575d3d6b9-registry-tls\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.174000 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.173930    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58690a93-436a-45f3-ab67-16d575d3d6b9-bound-sa-token\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.174000 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.173964    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58690a93-436a-45f3-ab67-16d575d3d6b9-image-registry-private-configuration\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.174343 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.174318    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58690a93-436a-45f3-ab67-16d575d3d6b9-ca-trust-extracted\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.174768 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.174746    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58690a93-436a-45f3-ab67-16d575d3d6b9-trusted-ca\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.174890 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.174869    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58690a93-436a-45f3-ab67-16d575d3d6b9-registry-certificates\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.176523 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.176501    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58690a93-436a-45f3-ab67-16d575d3d6b9-installation-pull-secrets\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.176601 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.176523    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58690a93-436a-45f3-ab67-16d575d3d6b9-image-registry-private-configuration\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.176601 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.176585    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58690a93-436a-45f3-ab67-16d575d3d6b9-registry-tls\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.188169 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.188144    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58690a93-436a-45f3-ab67-16d575d3d6b9-bound-sa-token\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.188365 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.188349    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xzq\" (UniqueName: \"kubernetes.io/projected/58690a93-436a-45f3-ab67-16d575d3d6b9-kube-api-access-98xzq\") pod \"image-registry-54f647d6c4-6hdhd\" (UID: \"58690a93-436a-45f3-ab67-16d575d3d6b9\") " pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.215595 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.215572    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"
Sep 17 21:59:46.310733 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.310703    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.494066 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.493991    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2"]
Sep 17 21:59:46.496700 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:46.496676    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8665ab3e_e814_4d38_a138_2f8c9a4a854e.slice/crio-328c582b316e1b5a2c0bf2875d2b13eca5101d507796cceb1ffeca2a8348f838 WatchSource:0}: Error finding container 328c582b316e1b5a2c0bf2875d2b13eca5101d507796cceb1ffeca2a8348f838: Status 404 returned error can't find the container with id 328c582b316e1b5a2c0bf2875d2b13eca5101d507796cceb1ffeca2a8348f838
Sep 17 21:59:46.582894 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.582868    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54f647d6c4-6hdhd"]
Sep 17 21:59:46.585714 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:46.585692    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58690a93_436a_45f3_ab67_16d575d3d6b9.slice/crio-bb44c8436883c5c06abe35c616fb5f502caaebe207168730623b8f1c4bdae017 WatchSource:0}: Error finding container bb44c8436883c5c06abe35c616fb5f502caaebe207168730623b8f1c4bdae017: Status 404 returned error can't find the container with id bb44c8436883c5c06abe35c616fb5f502caaebe207168730623b8f1c4bdae017
Sep 17 21:59:46.829294 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.829257    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2" event={"ID":"8665ab3e-e814-4d38-a138-2f8c9a4a854e","Type":"ContainerStarted","Data":"328c582b316e1b5a2c0bf2875d2b13eca5101d507796cceb1ffeca2a8348f838"}
Sep 17 21:59:46.830849 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.830819    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd" event={"ID":"58690a93-436a-45f3-ab67-16d575d3d6b9","Type":"ContainerStarted","Data":"52c1d824cd8dac475021f1f755c49a2fc54c792e5378fc49717cb08a4afd35c2"}
Sep 17 21:59:46.830992 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.830912    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd" event={"ID":"58690a93-436a-45f3-ab67-16d575d3d6b9","Type":"ContainerStarted","Data":"bb44c8436883c5c06abe35c616fb5f502caaebe207168730623b8f1c4bdae017"}
Sep 17 21:59:46.830992 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.830976    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 21:59:46.853397 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:46.853344    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd" podStartSLOduration=1.853327298 podStartE2EDuration="1.853327298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:46.85212235 +0000 UTC m=+88.714375829" watchObservedRunningTime="2026-09-17 21:59:46.853327298 +0000 UTC m=+88.715580774"
Sep 17 21:59:48.293096 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:48.293058    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:48.296116 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:48.296089    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/43882037-9994-4040-a7cd-bc334661ace1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdd9f\" (UID: \"43882037-9994-4040-a7cd-bc334661ace1\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:48.542716 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:48.542532    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xdgfh\""
Sep 17 21:59:48.550540 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:48.550474    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"
Sep 17 21:59:48.842865 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:48.842789    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f"]
Sep 17 21:59:49.896040 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:49.896006    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43882037_9994_4040_a7cd_bc334661ace1.slice/crio-91f74e0454664d18712b01b003a9455cf4f75677b2e58e153f1f468cf4a58182 WatchSource:0}: Error finding container 91f74e0454664d18712b01b003a9455cf4f75677b2e58e153f1f468cf4a58182: Status 404 returned error can't find the container with id 91f74e0454664d18712b01b003a9455cf4f75677b2e58e153f1f468cf4a58182
Sep 17 21:59:50.847932 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:50.847863    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2" event={"ID":"8665ab3e-e814-4d38-a138-2f8c9a4a854e","Type":"ContainerStarted","Data":"ab5cde649bca4568ca12b7180f39e89c041e22d52936944035548d8e1f92ad79"}
Sep 17 21:59:50.848160 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:50.847951    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2" event={"ID":"8665ab3e-e814-4d38-a138-2f8c9a4a854e","Type":"ContainerStarted","Data":"22c826a9e4a330618c71ac4deaaf0bbf93c1ca0cf94b4379bf30d0312692f253"}
Sep 17 21:59:50.848160 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:50.847974    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2" event={"ID":"8665ab3e-e814-4d38-a138-2f8c9a4a854e","Type":"ContainerStarted","Data":"8e55c2c06b5c4317836957f45a1d6b7128696bd3982fcc8f720e04e7ae1ad767"}
Sep 17 21:59:50.849034 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:50.849008    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f" event={"ID":"43882037-9994-4040-a7cd-bc334661ace1","Type":"ContainerStarted","Data":"91f74e0454664d18712b01b003a9455cf4f75677b2e58e153f1f468cf4a58182"}
Sep 17 21:59:50.867781 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:50.867730    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-74dc4f586c-p4bz2" podStartSLOduration=2.177631748 podStartE2EDuration="5.867713177s" totalImagesPullingTime="3.690081429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:50.867633809 +0000 UTC m=+92.729887290" watchObservedRunningTime="2026-09-17 21:59:50.867713177 +0000 UTC m=+92.729966654"
Sep 17 21:59:51.853852 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:51.853817    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f" event={"ID":"43882037-9994-4040-a7cd-bc334661ace1","Type":"ContainerStarted","Data":"8451c98af204dea91f12abd007a2e82e9d41c75b178c9841c2ca39accee2121f"}
Sep 17 21:59:51.875312 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:51.875265    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdd9f" podStartSLOduration=34.265441668 podStartE2EDuration="35.875248786s" totalImagesPullingTime="1.609807118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:51.873765592 +0000 UTC m=+93.736019079" watchObservedRunningTime="2026-09-17 21:59:51.875248786 +0000 UTC m=+93.737502261"
Sep 17 21:59:57.758143 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:57.758112    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m2r9c"
Sep 17 21:59:58.067505 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:58.067474    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:59:58.069855 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:58.069831    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0157d6a-2e6f-4af0-88ad-842ddc9003ed-metrics-tls\") pod \"dns-default-z5rqh\" (UID: \"a0157d6a-2e6f-4af0-88ad-842ddc9003ed\") " pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:59:58.361243 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:58.361174    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb92b\""
Sep 17 21:59:58.368278 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:58.368256    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5rqh"
Sep 17 21:59:58.639336 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:58.639270    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5rqh"]
Sep 17 21:59:58.642148 ip-10-0-137-232 kubenswrapper[2585]: W0917 21:59:58.642123    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0157d6a_2e6f_4af0_88ad_842ddc9003ed.slice/crio-790441efcca97db9731958b4bba2707e38addae447d0e7deefc00d803647d304 WatchSource:0}: Error finding container 790441efcca97db9731958b4bba2707e38addae447d0e7deefc00d803647d304: Status 404 returned error can't find the container with id 790441efcca97db9731958b4bba2707e38addae447d0e7deefc00d803647d304
Sep 17 21:59:58.877252 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:58.877223    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5rqh" event={"ID":"a0157d6a-2e6f-4af0-88ad-842ddc9003ed","Type":"ContainerStarted","Data":"790441efcca97db9731958b4bba2707e38addae447d0e7deefc00d803647d304"}
Sep 17 21:59:59.454184 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.454152    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rvtl6"]
Sep 17 21:59:59.457255 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.457232    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.459833 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.459805    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 21:59:59.459966 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.459939    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 21:59:59.460137 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.460093    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 21:59:59.460260 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.460167    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 21:59:59.460898 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.460878    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krpp4\""
Sep 17 21:59:59.578254 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.578216    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-metrics-client-ca\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.578254 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.578249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.578522 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.578301    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.578522 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.578345    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-sys\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.578522 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.578369    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-wtmp\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.578522 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.578392    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdjf\" (UniqueName: \"kubernetes.io/projected/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-kube-api-access-drdjf\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.578736 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.578511    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-textfile\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.578736 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.578599    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-tls\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.578736 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.578635    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-root\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679161 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679117    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679344 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679183    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-sys\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679344 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679221    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-wtmp\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679344 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679247    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdjf\" (UniqueName: \"kubernetes.io/projected/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-kube-api-access-drdjf\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679344 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679277    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-textfile\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679344 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679310    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-tls\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679344 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679340    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-root\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679663 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679417    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-metrics-client-ca\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679663 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679663 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679474    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-sys\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679663 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679421    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-wtmp\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679663 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-root\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.679832 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.679809    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-textfile\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.680113 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.680084    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-metrics-client-ca\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.680237 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.680089    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.681860 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.681829    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.682123 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.682104    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-node-exporter-tls\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.687672 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.687649    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdjf\" (UniqueName: \"kubernetes.io/projected/6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2-kube-api-access-drdjf\") pod \"node-exporter-rvtl6\" (UID: \"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2\") " pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 21:59:59.768164 ip-10-0-137-232 kubenswrapper[2585]: I0917 21:59:59.768095    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvtl6"
Sep 17 22:00:00.040970 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:00:00.040935    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c529f35_bbf5_4ecb_b5fc_4f43a15f07e2.slice/crio-357983f69ad803d01ae19443122216decfe31dd6adb655a96acddce8a64deb3f WatchSource:0}: Error finding container 357983f69ad803d01ae19443122216decfe31dd6adb655a96acddce8a64deb3f: Status 404 returned error can't find the container with id 357983f69ad803d01ae19443122216decfe31dd6adb655a96acddce8a64deb3f
Sep 17 22:00:00.886096 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:00.886060    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5rqh" event={"ID":"a0157d6a-2e6f-4af0-88ad-842ddc9003ed","Type":"ContainerStarted","Data":"f641aef714d9bf0815ab7fb839094cf31c4c0c177644a23e93c7549bc8170c10"}
Sep 17 22:00:00.886298 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:00.886126    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5rqh" event={"ID":"a0157d6a-2e6f-4af0-88ad-842ddc9003ed","Type":"ContainerStarted","Data":"9a4f3d1a30320c8190a1c053e0f11c380f5e243b0800f2bb7a0a43e6d8747f6e"}
Sep 17 22:00:00.886298 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:00.886217    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z5rqh"
Sep 17 22:00:00.887507 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:00.887475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvtl6" event={"ID":"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2","Type":"ContainerStarted","Data":"357983f69ad803d01ae19443122216decfe31dd6adb655a96acddce8a64deb3f"}
Sep 17 22:00:00.904794 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:00.904753    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z5rqh" podStartSLOduration=65.46674681 podStartE2EDuration="1m6.904736721s" totalImagesPullingTime="1.437989911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:00.904582346 +0000 UTC m=+102.766835822" watchObservedRunningTime="2026-09-17 22:00:00.904736721 +0000 UTC m=+102.766990197"
Sep 17 22:00:01.892779 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:01.892745    2585 generic.go:356] "Generic (PLEG): container finished" podID="6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2" containerID="a4f540c9320057ec9f1259ee2f675f763b40d657dfc8b4f4a34afd58913361a4" exitCode=0
Sep 17 22:00:01.893128 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:01.892822    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvtl6" event={"ID":"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2","Type":"ContainerDied","Data":"a4f540c9320057ec9f1259ee2f675f763b40d657dfc8b4f4a34afd58913361a4"}
Sep 17 22:00:02.898170 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:02.898136    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvtl6" event={"ID":"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2","Type":"ContainerStarted","Data":"1a5ef0dd5d04fa6dee1066900419f46830853d7ae6511a20f0aceb12b32eb2af"}
Sep 17 22:00:02.898656 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:02.898184    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvtl6" event={"ID":"6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2","Type":"ContainerStarted","Data":"5bd812b99d20824fe1b4044bc10d34abd497305b052a863352ad3da647549bbe"}
Sep 17 22:00:02.917674 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:02.917627    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rvtl6" podStartSLOduration=2.974171415 podStartE2EDuration="3.917612849s" totalImagesPullingTime="876.868498ms" totalInitContainerRuntime="66.572936ms" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:02.916175664 +0000 UTC m=+104.778429140" watchObservedRunningTime="2026-09-17 22:00:02.917612849 +0000 UTC m=+104.779866325"
Sep 17 22:00:06.315319 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:06.315286    2585 patch_prober.go:28] interesting pod/image-registry-54f647d6c4-6hdhd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 22:00:06.315772 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:06.315335    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd" podUID="58690a93-436a-45f3-ab67-16d575d3d6b9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:00:07.839370 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:07.839343    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54f647d6c4-6hdhd"
Sep 17 22:00:10.894688 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:10.894658    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z5rqh"
Sep 17 22:00:11.467166 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.467128    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5446578f87-76dsj"]
Sep 17 22:00:11.470439 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.470401    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.472921 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.472895    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 22:00:11.473666 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.473621    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 22:00:11.473666 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.473659    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5m25g\""
Sep 17 22:00:11.473868 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.473859    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 22:00:11.473970 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.473874    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 22:00:11.474023 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.474007    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 22:00:11.474075 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.474007    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 22:00:11.474075 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.474039    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 22:00:11.479332 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.479311    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 22:00:11.482880 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.482859    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5446578f87-76dsj"]
Sep 17 22:00:11.564289 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.564262    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jk6l\" (UniqueName: \"kubernetes.io/projected/9b2b566f-70c6-4f93-a5fe-600e96623894-kube-api-access-7jk6l\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.564423 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.564308    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-console-config\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.564423 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.564329    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-trusted-ca-bundle\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.564423 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.564352    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-oauth-serving-cert\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.564423 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.564414    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-oauth-config\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.564579 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.564447    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-service-ca\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.564579 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.564478    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-serving-cert\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.665592 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.665556    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-oauth-config\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.665592 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.665596    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-service-ca\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.665755 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.665620    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-serving-cert\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.665755 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.665636    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jk6l\" (UniqueName: \"kubernetes.io/projected/9b2b566f-70c6-4f93-a5fe-600e96623894-kube-api-access-7jk6l\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.665755 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.665660    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-console-config\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.665907 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.665777    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-trusted-ca-bundle\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.665907 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.665828    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-oauth-serving-cert\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.666400 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.666377    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-console-config\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.666511 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.666468    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-oauth-serving-cert\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.666576 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.666471    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-service-ca\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.666740 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.666723    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-trusted-ca-bundle\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.668203 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.668183    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-oauth-config\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.668311 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.668292    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-serving-cert\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.674563 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.674545    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jk6l\" (UniqueName: \"kubernetes.io/projected/9b2b566f-70c6-4f93-a5fe-600e96623894-kube-api-access-7jk6l\") pod \"console-5446578f87-76dsj\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") " pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:11.780614 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:11.780592    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:12.051549 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:12.051482    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5446578f87-76dsj"]
Sep 17 22:00:12.053962 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:00:12.053936    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2b566f_70c6_4f93_a5fe_600e96623894.slice/crio-b9714b98ac769bd7dffd37081fc4b3440dfcb3b33f56797bbadb86f550ca7a0a WatchSource:0}: Error finding container b9714b98ac769bd7dffd37081fc4b3440dfcb3b33f56797bbadb86f550ca7a0a: Status 404 returned error can't find the container with id b9714b98ac769bd7dffd37081fc4b3440dfcb3b33f56797bbadb86f550ca7a0a
Sep 17 22:00:12.940902 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:12.940867    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5446578f87-76dsj" event={"ID":"9b2b566f-70c6-4f93-a5fe-600e96623894","Type":"ContainerStarted","Data":"b9714b98ac769bd7dffd37081fc4b3440dfcb3b33f56797bbadb86f550ca7a0a"}
Sep 17 22:00:16.958231 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:16.958196    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5446578f87-76dsj" event={"ID":"9b2b566f-70c6-4f93-a5fe-600e96623894","Type":"ContainerStarted","Data":"a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490"}
Sep 17 22:00:16.977243 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:16.977195    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5446578f87-76dsj" podStartSLOduration=2.121768728 podStartE2EDuration="5.977150984s" totalImagesPullingTime="3.855382256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:16.976269685 +0000 UTC m=+118.838523162" watchObservedRunningTime="2026-09-17 22:00:16.977150984 +0000 UTC m=+118.839404460"
Sep 17 22:00:21.780872 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:21.780835    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:21.781351 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:21.780895    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:21.785892 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:21.785869    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:21.980056 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:21.980026    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:00:25.991193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:25.991113    2585 generic.go:356] "Generic (PLEG): container finished" podID="7b514b24-b573-493e-bd63-ff52b7f4a787" containerID="52587c7159d1b5382ea71a7bd629b020bd824703fbb924eb2283a948d4ee9c88" exitCode=0
Sep 17 22:00:25.991193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:25.991160    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm" event={"ID":"7b514b24-b573-493e-bd63-ff52b7f4a787","Type":"ContainerDied","Data":"52587c7159d1b5382ea71a7bd629b020bd824703fbb924eb2283a948d4ee9c88"}
Sep 17 22:00:25.991647 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:25.991457    2585 scope.go:117] "RemoveContainer" containerID="52587c7159d1b5382ea71a7bd629b020bd824703fbb924eb2283a948d4ee9c88"
Sep 17 22:00:27.000407 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:27.000371    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-mh7vm" event={"ID":"7b514b24-b573-493e-bd63-ff52b7f4a787","Type":"ContainerStarted","Data":"9dfb6887f8050ce8e8994c8eae340db9fb6cf95c3e1a3b8318ee9e45b33d6b91"}
Sep 17 22:00:30.312410 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:30.312375    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 22:00:30.314827 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:30.314807    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a3a05c-bf13-4011-a95e-36271b66ec5d-metrics-certs\") pod \"network-metrics-daemon-lm5n6\" (UID: \"12a3a05c-bf13-4011-a95e-36271b66ec5d\") " pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 22:00:30.473072 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:30.473044    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pzq2x\""
Sep 17 22:00:30.481080 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:30.481061    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lm5n6"
Sep 17 22:00:30.748855 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:30.748787    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lm5n6"]
Sep 17 22:00:30.751966 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:00:30.751941    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a3a05c_bf13_4011_a95e_36271b66ec5d.slice/crio-925c0d9feb641e156157b9a6c154282d9a5eb087f7a3863a17f8c4c089ca442f WatchSource:0}: Error finding container 925c0d9feb641e156157b9a6c154282d9a5eb087f7a3863a17f8c4c089ca442f: Status 404 returned error can't find the container with id 925c0d9feb641e156157b9a6c154282d9a5eb087f7a3863a17f8c4c089ca442f
Sep 17 22:00:31.014682 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:31.014599    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lm5n6" event={"ID":"12a3a05c-bf13-4011-a95e-36271b66ec5d","Type":"ContainerStarted","Data":"925c0d9feb641e156157b9a6c154282d9a5eb087f7a3863a17f8c4c089ca442f"}
Sep 17 22:00:32.019891 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:32.019857    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lm5n6" event={"ID":"12a3a05c-bf13-4011-a95e-36271b66ec5d","Type":"ContainerStarted","Data":"065ee6a570066f2841926daac7a239572062adc5c3cde81b3b8379cc4e3e4be1"}
Sep 17 22:00:32.020319 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:32.019917    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lm5n6" event={"ID":"12a3a05c-bf13-4011-a95e-36271b66ec5d","Type":"ContainerStarted","Data":"d6cd9580b56b35194ddbe24b49819658ffac6c9a4fc3807b3f28adddbda7c150"}
Sep 17 22:00:32.036495 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:00:32.036449    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lm5n6" podStartSLOduration=131.132797483 podStartE2EDuration="2m12.036414162s" totalImagesPullingTime="903.616679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:32.03547843 +0000 UTC m=+133.897731908" watchObservedRunningTime="2026-09-17 22:00:32.036414162 +0000 UTC m=+133.898667637"
Sep 17 22:01:26.781628 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.781550    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65f89c9979-vkp87"]
Sep 17 22:01:26.784894 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.784876    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.794422 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.794400    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65f89c9979-vkp87"]
Sep 17 22:01:26.884131 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.884107    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xmk\" (UniqueName: \"kubernetes.io/projected/2bdbce1a-9fef-4f74-a656-e4b014d8a031-kube-api-access-x6xmk\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.884261 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.884136    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-config\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.884261 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.884158    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-trusted-ca-bundle\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.884383 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.884258    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-oauth-config\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.884383 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.884290    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-service-ca\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.884383 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.884335    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-serving-cert\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.884529 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.884387    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-oauth-serving-cert\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.985509 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.985475    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-oauth-config\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.985638 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.985512    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-service-ca\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.985638 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.985546    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-serving-cert\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.985638 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.985592    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-oauth-serving-cert\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.985638 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.985633    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xmk\" (UniqueName: \"kubernetes.io/projected/2bdbce1a-9fef-4f74-a656-e4b014d8a031-kube-api-access-x6xmk\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.985832 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.985662    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-config\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.985881 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.985843    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-trusted-ca-bundle\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.986255 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.986229    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-service-ca\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.986358 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.986292    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-oauth-serving-cert\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.986593 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.986573    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-config\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.987150 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.987127    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-trusted-ca-bundle\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.988272 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.988241    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-oauth-config\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.988371 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.988354    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-serving-cert\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:26.994125 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:26.994105    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xmk\" (UniqueName: \"kubernetes.io/projected/2bdbce1a-9fef-4f74-a656-e4b014d8a031-kube-api-access-x6xmk\") pod \"console-65f89c9979-vkp87\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") " pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:27.093659 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:27.093616    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:27.365506 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:27.365441    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65f89c9979-vkp87"]
Sep 17 22:01:27.368152 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:01:27.368124    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdbce1a_9fef_4f74_a656_e4b014d8a031.slice/crio-3d891e9e2f151095535e2981dda0edcabf9d12ae0638c6632a45d16ee963c024 WatchSource:0}: Error finding container 3d891e9e2f151095535e2981dda0edcabf9d12ae0638c6632a45d16ee963c024: Status 404 returned error can't find the container with id 3d891e9e2f151095535e2981dda0edcabf9d12ae0638c6632a45d16ee963c024
Sep 17 22:01:28.225940 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:28.225911    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f89c9979-vkp87" event={"ID":"2bdbce1a-9fef-4f74-a656-e4b014d8a031","Type":"ContainerStarted","Data":"03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8"}
Sep 17 22:01:28.226326 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:28.225957    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f89c9979-vkp87" event={"ID":"2bdbce1a-9fef-4f74-a656-e4b014d8a031","Type":"ContainerStarted","Data":"3d891e9e2f151095535e2981dda0edcabf9d12ae0638c6632a45d16ee963c024"}
Sep 17 22:01:28.245505 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:28.245461    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65f89c9979-vkp87" podStartSLOduration=2.245441216 podStartE2EDuration="2.245441216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:01:28.243675643 +0000 UTC m=+190.105929121" watchObservedRunningTime="2026-09-17 22:01:28.245441216 +0000 UTC m=+190.107694684"
Sep 17 22:01:37.094653 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:37.094616    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:37.095133 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:37.094685    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:37.099115 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:37.099096    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:37.261165 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:37.261136    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:01:37.302159 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:01:37.302124    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5446578f87-76dsj"]
Sep 17 22:02:02.326165 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.326121    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5446578f87-76dsj" podUID="9b2b566f-70c6-4f93-a5fe-600e96623894" containerName="console" containerID="cri-o://a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490" gracePeriod=15
Sep 17 22:02:02.578286 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.578227    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:02:02.737143 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737107    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-trusted-ca-bundle\") pod \"9b2b566f-70c6-4f93-a5fe-600e96623894\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") "
Sep 17 22:02:02.737143 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737152    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-oauth-config\") pod \"9b2b566f-70c6-4f93-a5fe-600e96623894\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") "
Sep 17 22:02:02.737385 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737197    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jk6l\" (UniqueName: \"kubernetes.io/projected/9b2b566f-70c6-4f93-a5fe-600e96623894-kube-api-access-7jk6l\") pod \"9b2b566f-70c6-4f93-a5fe-600e96623894\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") "
Sep 17 22:02:02.737385 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737216    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-oauth-serving-cert\") pod \"9b2b566f-70c6-4f93-a5fe-600e96623894\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") "
Sep 17 22:02:02.737385 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737236    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-console-config\") pod \"9b2b566f-70c6-4f93-a5fe-600e96623894\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") "
Sep 17 22:02:02.737385 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737265    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-serving-cert\") pod \"9b2b566f-70c6-4f93-a5fe-600e96623894\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") "
Sep 17 22:02:02.737385 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737298    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-service-ca\") pod \"9b2b566f-70c6-4f93-a5fe-600e96623894\" (UID: \"9b2b566f-70c6-4f93-a5fe-600e96623894\") "
Sep 17 22:02:02.737754 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737722    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-console-config" (OuterVolumeSpecName: "console-config") pod "9b2b566f-70c6-4f93-a5fe-600e96623894" (UID: "9b2b566f-70c6-4f93-a5fe-600e96623894"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:02.737861 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737757    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9b2b566f-70c6-4f93-a5fe-600e96623894" (UID: "9b2b566f-70c6-4f93-a5fe-600e96623894"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:02.737861 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737774    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b2b566f-70c6-4f93-a5fe-600e96623894" (UID: "9b2b566f-70c6-4f93-a5fe-600e96623894"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:02.737861 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.737785    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b2b566f-70c6-4f93-a5fe-600e96623894" (UID: "9b2b566f-70c6-4f93-a5fe-600e96623894"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:02:02.739800 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.739768    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b2b566f-70c6-4f93-a5fe-600e96623894" (UID: "9b2b566f-70c6-4f93-a5fe-600e96623894"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:02:02.739903 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.739846    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b2b566f-70c6-4f93-a5fe-600e96623894" (UID: "9b2b566f-70c6-4f93-a5fe-600e96623894"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:02:02.739903 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.739851    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2b566f-70c6-4f93-a5fe-600e96623894-kube-api-access-7jk6l" (OuterVolumeSpecName: "kube-api-access-7jk6l") pod "9b2b566f-70c6-4f93-a5fe-600e96623894" (UID: "9b2b566f-70c6-4f93-a5fe-600e96623894"). InnerVolumeSpecName "kube-api-access-7jk6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:02:02.838218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.838165    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jk6l\" (UniqueName: \"kubernetes.io/projected/9b2b566f-70c6-4f93-a5fe-600e96623894-kube-api-access-7jk6l\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:02.838218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.838188    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-oauth-serving-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:02.838218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.838197    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-console-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:02.838218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.838206    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-serving-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:02.838218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.838213    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-service-ca\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:02.838218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.838221    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b2b566f-70c6-4f93-a5fe-600e96623894-trusted-ca-bundle\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:02.838472 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:02.838229    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b2b566f-70c6-4f93-a5fe-600e96623894-console-oauth-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:03.348739 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:03.348704    2585 generic.go:356] "Generic (PLEG): container finished" podID="9b2b566f-70c6-4f93-a5fe-600e96623894" containerID="a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490" exitCode=0
Sep 17 22:02:03.349125 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:03.348754    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5446578f87-76dsj"
Sep 17 22:02:03.349125 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:03.348791    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5446578f87-76dsj" event={"ID":"9b2b566f-70c6-4f93-a5fe-600e96623894","Type":"ContainerDied","Data":"a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490"}
Sep 17 22:02:03.349125 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:03.348843    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5446578f87-76dsj" event={"ID":"9b2b566f-70c6-4f93-a5fe-600e96623894","Type":"ContainerDied","Data":"b9714b98ac769bd7dffd37081fc4b3440dfcb3b33f56797bbadb86f550ca7a0a"}
Sep 17 22:02:03.349125 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:03.348867    2585 scope.go:117] "RemoveContainer" containerID="a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490"
Sep 17 22:02:03.362044 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:03.362015    2585 scope.go:117] "RemoveContainer" containerID="a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490"
Sep 17 22:02:03.362383 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:02:03.362359    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490\": container with ID starting with a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490 not found: ID does not exist" containerID="a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490"
Sep 17 22:02:03.362480 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:03.362390    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490"} err="failed to get container status \"a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490\": rpc error: code = NotFound desc = could not find container \"a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490\": container with ID starting with a2c132dfac32cf68adff45f2d1e7e84323ea973b361f916a6d9c4949428b7490 not found: ID does not exist"
Sep 17 22:02:03.371486 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:03.371463    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5446578f87-76dsj"]
Sep 17 22:02:03.376939 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:03.376917    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5446578f87-76dsj"]
Sep 17 22:02:04.463009 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:04.462979    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b2b566f-70c6-4f93-a5fe-600e96623894" path="/var/lib/kubelet/pods/9b2b566f-70c6-4f93-a5fe-600e96623894/volumes"
Sep 17 22:02:38.706929 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.706850    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"]
Sep 17 22:02:38.707343 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.707093    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2b566f-70c6-4f93-a5fe-600e96623894" containerName="console"
Sep 17 22:02:38.707343 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.707104    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2b566f-70c6-4f93-a5fe-600e96623894" containerName="console"
Sep 17 22:02:38.707343 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.707144    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b2b566f-70c6-4f93-a5fe-600e96623894" containerName="console"
Sep 17 22:02:38.710029 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.710012    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:38.712484 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.712461    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 22:02:38.713362 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.713343    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dfxzr\""
Sep 17 22:02:38.713362 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.713359    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 22:02:38.718760 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.718733    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"]
Sep 17 22:02:38.768090 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.768067    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:38.768183 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.768113    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tlh\" (UniqueName: \"kubernetes.io/projected/01f50a11-7ce4-43a7-a867-0ea6acffd21c-kube-api-access-q9tlh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:38.768183 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.768164    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:38.868886 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.868864    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:38.868964 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.868906    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tlh\" (UniqueName: \"kubernetes.io/projected/01f50a11-7ce4-43a7-a867-0ea6acffd21c-kube-api-access-q9tlh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:38.868964 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.868932    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:38.869195 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.869177    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:38.869232 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.869217    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:38.876533 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:38.876514    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tlh\" (UniqueName: \"kubernetes.io/projected/01f50a11-7ce4-43a7-a867-0ea6acffd21c-kube-api-access-q9tlh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:39.019030 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:39.018962    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:39.290563 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:39.290353    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"]
Sep 17 22:02:39.292879 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:02:39.292853    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f50a11_7ce4_43a7_a867_0ea6acffd21c.slice/crio-ac6b584efef1674661019dd2f1d842ea856be40a10d0a168d0fff8f35b1802cc WatchSource:0}: Error finding container ac6b584efef1674661019dd2f1d842ea856be40a10d0a168d0fff8f35b1802cc: Status 404 returned error can't find the container with id ac6b584efef1674661019dd2f1d842ea856be40a10d0a168d0fff8f35b1802cc
Sep 17 22:02:39.474162 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:39.474128    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6" event={"ID":"01f50a11-7ce4-43a7-a867-0ea6acffd21c","Type":"ContainerStarted","Data":"ac6b584efef1674661019dd2f1d842ea856be40a10d0a168d0fff8f35b1802cc"}
Sep 17 22:02:45.497473 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:45.497444    2585 generic.go:356] "Generic (PLEG): container finished" podID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerID="72b04996d7c83157dccaf507ba8e700a91dc61e9ac54f98bee92d458c23de73f" exitCode=0
Sep 17 22:02:45.497791 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:45.497502    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6" event={"ID":"01f50a11-7ce4-43a7-a867-0ea6acffd21c","Type":"ContainerDied","Data":"72b04996d7c83157dccaf507ba8e700a91dc61e9ac54f98bee92d458c23de73f"}
Sep 17 22:02:47.507747 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:47.507679    2585 generic.go:356] "Generic (PLEG): container finished" podID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerID="89dec2385a4347360c352a90b8a1c9554f426fdbaa27bfd44bd9956e19ebdfb0" exitCode=0
Sep 17 22:02:47.507747 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:47.507734    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6" event={"ID":"01f50a11-7ce4-43a7-a867-0ea6acffd21c","Type":"ContainerDied","Data":"89dec2385a4347360c352a90b8a1c9554f426fdbaa27bfd44bd9956e19ebdfb0"}
Sep 17 22:02:54.539787 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:54.539756    2585 generic.go:356] "Generic (PLEG): container finished" podID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerID="1bcdb0c986877e7c6c4c2e638b2a859d35498685249920db15a3c19676df966f" exitCode=0
Sep 17 22:02:54.540093 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:54.539847    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6" event={"ID":"01f50a11-7ce4-43a7-a867-0ea6acffd21c","Type":"ContainerDied","Data":"1bcdb0c986877e7c6c4c2e638b2a859d35498685249920db15a3c19676df966f"}
Sep 17 22:02:55.665584 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.665562    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:55.697081 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.697060    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-bundle\") pod \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") "
Sep 17 22:02:55.697204 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.697105    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9tlh\" (UniqueName: \"kubernetes.io/projected/01f50a11-7ce4-43a7-a867-0ea6acffd21c-kube-api-access-q9tlh\") pod \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") "
Sep 17 22:02:55.697204 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.697127    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-util\") pod \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\" (UID: \"01f50a11-7ce4-43a7-a867-0ea6acffd21c\") "
Sep 17 22:02:55.697660 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.697638    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-bundle" (OuterVolumeSpecName: "bundle") pod "01f50a11-7ce4-43a7-a867-0ea6acffd21c" (UID: "01f50a11-7ce4-43a7-a867-0ea6acffd21c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:02:55.699450 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.699410    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f50a11-7ce4-43a7-a867-0ea6acffd21c-kube-api-access-q9tlh" (OuterVolumeSpecName: "kube-api-access-q9tlh") pod "01f50a11-7ce4-43a7-a867-0ea6acffd21c" (UID: "01f50a11-7ce4-43a7-a867-0ea6acffd21c"). InnerVolumeSpecName "kube-api-access-q9tlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:02:55.703265 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.703244    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-util" (OuterVolumeSpecName: "util") pod "01f50a11-7ce4-43a7-a867-0ea6acffd21c" (UID: "01f50a11-7ce4-43a7-a867-0ea6acffd21c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:02:55.798291 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.798231    2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-bundle\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:55.798291 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.798263    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9tlh\" (UniqueName: \"kubernetes.io/projected/01f50a11-7ce4-43a7-a867-0ea6acffd21c-kube-api-access-q9tlh\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:55.798291 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:55.798273    2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01f50a11-7ce4-43a7-a867-0ea6acffd21c-util\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:02:56.548553 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:56.548524    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6"
Sep 17 22:02:56.548553 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:56.548540    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alctz6" event={"ID":"01f50a11-7ce4-43a7-a867-0ea6acffd21c","Type":"ContainerDied","Data":"ac6b584efef1674661019dd2f1d842ea856be40a10d0a168d0fff8f35b1802cc"}
Sep 17 22:02:56.548768 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:02:56.548577    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac6b584efef1674661019dd2f1d842ea856be40a10d0a168d0fff8f35b1802cc"
Sep 17 22:03:06.003215 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.003184    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-j6fn7"]
Sep 17 22:03:06.003591 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.003470    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerName="util"
Sep 17 22:03:06.003591 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.003482    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerName="util"
Sep 17 22:03:06.003591 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.003490    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerName="pull"
Sep 17 22:03:06.003591 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.003495    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerName="pull"
Sep 17 22:03:06.003591 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.003501    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerName="extract"
Sep 17 22:03:06.003591 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.003508    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerName="extract"
Sep 17 22:03:06.003591 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.003567    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f50a11-7ce4-43a7-a867-0ea6acffd21c" containerName="extract"
Sep 17 22:03:06.018283 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.018262    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.020444 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.020404    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-j6fn7"]
Sep 17 22:03:06.020835 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.020810    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 22:03:06.020931 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.020810    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 22:03:06.020931 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.020871    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-64shg\""
Sep 17 22:03:06.020931 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.020924    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 22:03:06.021792 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.021774    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 22:03:06.021792 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.021786    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 22:03:06.066267 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.066241    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/32026b6d-92c2-499b-a33a-ba973be5bef1-cabundle0\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.066365 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.066269    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrhf\" (UniqueName: \"kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-kube-api-access-nqrhf\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.066365 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.066299    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-certificates\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.167389 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.167355    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/32026b6d-92c2-499b-a33a-ba973be5bef1-cabundle0\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.167567 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.167399    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrhf\" (UniqueName: \"kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-kube-api-access-nqrhf\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.167567 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.167466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-certificates\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.167688 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:03:06.167593    2585 secret.go:281] references non-existent secret key: ca.crt
Sep 17 22:03:06.167688 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:03:06.167610    2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 22:03:06.167688 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:03:06.167621    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-j6fn7: references non-existent secret key: ca.crt
Sep 17 22:03:06.167688 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:03:06.167684    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-certificates podName:32026b6d-92c2-499b-a33a-ba973be5bef1 nodeName:}" failed. No retries permitted until 2026-09-17 22:03:06.667665343 +0000 UTC m=+288.529918811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-certificates") pod "keda-operator-579c8958f7-j6fn7" (UID: "32026b6d-92c2-499b-a33a-ba973be5bef1") : references non-existent secret key: ca.crt
Sep 17 22:03:06.168108 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.168085    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/32026b6d-92c2-499b-a33a-ba973be5bef1-cabundle0\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.182793 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.182772    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrhf\" (UniqueName: \"kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-kube-api-access-nqrhf\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.670628 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:06.670592    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-certificates\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:06.670957 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:03:06.670731    2585 secret.go:281] references non-existent secret key: ca.crt
Sep 17 22:03:06.670957 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:03:06.670759    2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 22:03:06.670957 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:03:06.670769    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-j6fn7: references non-existent secret key: ca.crt
Sep 17 22:03:06.670957 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:03:06.670829    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-certificates podName:32026b6d-92c2-499b-a33a-ba973be5bef1 nodeName:}" failed. No retries permitted until 2026-09-17 22:03:07.670814439 +0000 UTC m=+289.533067893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-certificates") pod "keda-operator-579c8958f7-j6fn7" (UID: "32026b6d-92c2-499b-a33a-ba973be5bef1") : references non-existent secret key: ca.crt
Sep 17 22:03:07.679935 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:07.679899    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-certificates\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:07.682733 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:07.682703    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/32026b6d-92c2-499b-a33a-ba973be5bef1-certificates\") pod \"keda-operator-579c8958f7-j6fn7\" (UID: \"32026b6d-92c2-499b-a33a-ba973be5bef1\") " pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:07.831379 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:07.831346    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:08.100396 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:08.100372    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-j6fn7"]
Sep 17 22:03:08.102703 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:03:08.102672    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32026b6d_92c2_499b_a33a_ba973be5bef1.slice/crio-c8597c5b188ae6bbbb8bb1897444241712b2db7dd3b45ec8ce165b266f0f99b0 WatchSource:0}: Error finding container c8597c5b188ae6bbbb8bb1897444241712b2db7dd3b45ec8ce165b266f0f99b0: Status 404 returned error can't find the container with id c8597c5b188ae6bbbb8bb1897444241712b2db7dd3b45ec8ce165b266f0f99b0
Sep 17 22:03:08.592058 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:08.592022    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-j6fn7" event={"ID":"32026b6d-92c2-499b-a33a-ba973be5bef1","Type":"ContainerStarted","Data":"c8597c5b188ae6bbbb8bb1897444241712b2db7dd3b45ec8ce165b266f0f99b0"}
Sep 17 22:03:11.605970 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:11.605933    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-j6fn7" event={"ID":"32026b6d-92c2-499b-a33a-ba973be5bef1","Type":"ContainerStarted","Data":"b9ae69fd2d73f27dab71a557f22d0c4ac3be2ffca8d70ffa0215268b903f3820"}
Sep 17 22:03:11.606312 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:11.606047    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:03:11.621358 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:11.621281    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-j6fn7" podStartSLOduration=3.359413299 podStartE2EDuration="6.621265631s" totalImagesPullingTime="3.261852332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:11.621034358 +0000 UTC m=+293.483287834" watchObservedRunningTime="2026-09-17 22:03:11.621265631 +0000 UTC m=+293.483519106"
Sep 17 22:03:20.350479 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:20.350452    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:03:20.352392 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:20.352368    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:03:20.354409 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:20.354390    2585 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 22:03:32.612232 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:03:32.612192    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-j6fn7"
Sep 17 22:04:23.205662 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.205577    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-t79z2"]
Sep 17 22:04:23.207959 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.207936    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:04:23.210304 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.210253    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 22:04:23.210458 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.210391    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:04:23.210529 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.210494    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z6gz2\""
Sep 17 22:04:23.211182 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.211169    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:04:23.215960 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.215929    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-t79z2"]
Sep 17 22:04:23.305097 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.305066    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gd7\" (UniqueName: \"kubernetes.io/projected/0ed0b33c-6cf3-476b-b263-852224b92a2f-kube-api-access-s9gd7\") pod \"seaweedfs-64568bcd49-t79z2\" (UID: \"0ed0b33c-6cf3-476b-b263-852224b92a2f\") " pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:04:23.305207 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.305124    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ed0b33c-6cf3-476b-b263-852224b92a2f-data\") pod \"seaweedfs-64568bcd49-t79z2\" (UID: \"0ed0b33c-6cf3-476b-b263-852224b92a2f\") " pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:04:23.406388 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.406364    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gd7\" (UniqueName: \"kubernetes.io/projected/0ed0b33c-6cf3-476b-b263-852224b92a2f-kube-api-access-s9gd7\") pod \"seaweedfs-64568bcd49-t79z2\" (UID: \"0ed0b33c-6cf3-476b-b263-852224b92a2f\") " pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:04:23.406505 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.406416    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ed0b33c-6cf3-476b-b263-852224b92a2f-data\") pod \"seaweedfs-64568bcd49-t79z2\" (UID: \"0ed0b33c-6cf3-476b-b263-852224b92a2f\") " pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:04:23.406737 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.406722    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ed0b33c-6cf3-476b-b263-852224b92a2f-data\") pod \"seaweedfs-64568bcd49-t79z2\" (UID: \"0ed0b33c-6cf3-476b-b263-852224b92a2f\") " pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:04:23.414565 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.414545    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gd7\" (UniqueName: \"kubernetes.io/projected/0ed0b33c-6cf3-476b-b263-852224b92a2f-kube-api-access-s9gd7\") pod \"seaweedfs-64568bcd49-t79z2\" (UID: \"0ed0b33c-6cf3-476b-b263-852224b92a2f\") " pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:04:23.517878 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.517829    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:04:23.788704 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.788677    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-t79z2"]
Sep 17 22:04:23.792421 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:04:23.792396    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed0b33c_6cf3_476b_b263_852224b92a2f.slice/crio-9b423fe518923e0ff1e4340a14e84319e146b387bce6120e13e5b6481d914ba7 WatchSource:0}: Error finding container 9b423fe518923e0ff1e4340a14e84319e146b387bce6120e13e5b6481d914ba7: Status 404 returned error can't find the container with id 9b423fe518923e0ff1e4340a14e84319e146b387bce6120e13e5b6481d914ba7
Sep 17 22:04:23.793766 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.793748    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:04:23.870774 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:23.870735    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-t79z2" event={"ID":"0ed0b33c-6cf3-476b-b263-852224b92a2f","Type":"ContainerStarted","Data":"9b423fe518923e0ff1e4340a14e84319e146b387bce6120e13e5b6481d914ba7"}
Sep 17 22:04:26.883482 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:26.883443    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-t79z2" event={"ID":"0ed0b33c-6cf3-476b-b263-852224b92a2f","Type":"ContainerStarted","Data":"7a618925e21525a3bc7f8a9d1ad255f74a933a06d516f49035b739c03438410f"}
Sep 17 22:04:26.883856 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:26.883579    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:04:26.899469 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:26.899412    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-t79z2" podStartSLOduration=1.251386194 podStartE2EDuration="3.899396185s" totalImagesPullingTime="2.648009991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:04:26.898284008 +0000 UTC m=+368.760537485" watchObservedRunningTime="2026-09-17 22:04:26.899396185 +0000 UTC m=+368.761649663"
Sep 17 22:04:32.889857 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:04:32.889829    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-t79z2"
Sep 17 22:05:05.046841 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:05.046811    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65f89c9979-vkp87"]
Sep 17 22:05:30.066073 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.066025    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65f89c9979-vkp87" podUID="2bdbce1a-9fef-4f74-a656-e4b014d8a031" containerName="console" containerID="cri-o://03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8" gracePeriod=15
Sep 17 22:05:30.304113 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.304089    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:05:30.373884 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.373819    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-trusted-ca-bundle\") pod \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") "
Sep 17 22:05:30.373884 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.373851    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-oauth-config\") pod \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") "
Sep 17 22:05:30.374073 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.373886    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6xmk\" (UniqueName: \"kubernetes.io/projected/2bdbce1a-9fef-4f74-a656-e4b014d8a031-kube-api-access-x6xmk\") pod \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") "
Sep 17 22:05:30.374073 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.373912    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-oauth-serving-cert\") pod \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") "
Sep 17 22:05:30.374073 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.374045    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-service-ca\") pod \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") "
Sep 17 22:05:30.374218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.374098    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-serving-cert\") pod \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") "
Sep 17 22:05:30.374218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.374139    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-config\") pod \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\" (UID: \"2bdbce1a-9fef-4f74-a656-e4b014d8a031\") "
Sep 17 22:05:30.374375 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.374350    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2bdbce1a-9fef-4f74-a656-e4b014d8a031" (UID: "2bdbce1a-9fef-4f74-a656-e4b014d8a031"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:30.374375 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.374359    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2bdbce1a-9fef-4f74-a656-e4b014d8a031" (UID: "2bdbce1a-9fef-4f74-a656-e4b014d8a031"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:30.374522 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.374403    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-service-ca" (OuterVolumeSpecName: "service-ca") pod "2bdbce1a-9fef-4f74-a656-e4b014d8a031" (UID: "2bdbce1a-9fef-4f74-a656-e4b014d8a031"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:30.374717 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.374697    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-config" (OuterVolumeSpecName: "console-config") pod "2bdbce1a-9fef-4f74-a656-e4b014d8a031" (UID: "2bdbce1a-9fef-4f74-a656-e4b014d8a031"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:05:30.376077 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.376054    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2bdbce1a-9fef-4f74-a656-e4b014d8a031" (UID: "2bdbce1a-9fef-4f74-a656-e4b014d8a031"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:05:30.376174 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.376130    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdbce1a-9fef-4f74-a656-e4b014d8a031-kube-api-access-x6xmk" (OuterVolumeSpecName: "kube-api-access-x6xmk") pod "2bdbce1a-9fef-4f74-a656-e4b014d8a031" (UID: "2bdbce1a-9fef-4f74-a656-e4b014d8a031"). InnerVolumeSpecName "kube-api-access-x6xmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:05:30.376251 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.376230    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2bdbce1a-9fef-4f74-a656-e4b014d8a031" (UID: "2bdbce1a-9fef-4f74-a656-e4b014d8a031"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:05:30.475038 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.475018    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-serving-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:05:30.475038 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.475039    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:05:30.475166 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.475048    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-trusted-ca-bundle\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:05:30.475166 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.475057    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdbce1a-9fef-4f74-a656-e4b014d8a031-console-oauth-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:05:30.475166 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.475066    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6xmk\" (UniqueName: \"kubernetes.io/projected/2bdbce1a-9fef-4f74-a656-e4b014d8a031-kube-api-access-x6xmk\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:05:30.475166 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.475075    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-oauth-serving-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:05:30.475166 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:30.475083    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdbce1a-9fef-4f74-a656-e4b014d8a031-service-ca\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:05:31.121897 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:31.121861    2585 generic.go:356] "Generic (PLEG): container finished" podID="2bdbce1a-9fef-4f74-a656-e4b014d8a031" containerID="03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8" exitCode=0
Sep 17 22:05:31.122317 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:31.121910    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65f89c9979-vkp87"
Sep 17 22:05:31.122317 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:31.121959    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f89c9979-vkp87" event={"ID":"2bdbce1a-9fef-4f74-a656-e4b014d8a031","Type":"ContainerDied","Data":"03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8"}
Sep 17 22:05:31.122317 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:31.122010    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65f89c9979-vkp87" event={"ID":"2bdbce1a-9fef-4f74-a656-e4b014d8a031","Type":"ContainerDied","Data":"3d891e9e2f151095535e2981dda0edcabf9d12ae0638c6632a45d16ee963c024"}
Sep 17 22:05:31.122317 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:31.122031    2585 scope.go:117] "RemoveContainer" containerID="03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8"
Sep 17 22:05:31.130508 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:31.130493    2585 scope.go:117] "RemoveContainer" containerID="03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8"
Sep 17 22:05:31.130776 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:05:31.130756    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8\": container with ID starting with 03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8 not found: ID does not exist" containerID="03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8"
Sep 17 22:05:31.130840 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:31.130788    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8"} err="failed to get container status \"03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8\": rpc error: code = NotFound desc = could not find container \"03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8\": container with ID starting with 03d9563da16b5b9d35bbafb822aa587a207da0d5b404d805662e8bd5ac4f10b8 not found: ID does not exist"
Sep 17 22:05:31.139313 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:31.139294    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65f89c9979-vkp87"]
Sep 17 22:05:31.143392 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:31.143372    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65f89c9979-vkp87"]
Sep 17 22:05:32.463343 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:32.463308    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdbce1a-9fef-4f74-a656-e4b014d8a031" path="/var/lib/kubelet/pods/2bdbce1a-9fef-4f74-a656-e4b014d8a031/volumes"
Sep 17 22:05:40.309247 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.307266    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-bjj8m"]
Sep 17 22:05:40.309247 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.307894    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdbce1a-9fef-4f74-a656-e4b014d8a031" containerName="console"
Sep 17 22:05:40.309247 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.307914    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdbce1a-9fef-4f74-a656-e4b014d8a031" containerName="console"
Sep 17 22:05:40.309247 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.308048    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdbce1a-9fef-4f74-a656-e4b014d8a031" containerName="console"
Sep 17 22:05:40.312595 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.312560    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:40.314843 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.314825    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 22:05:40.314956 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.314915    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-c45bh\""
Sep 17 22:05:40.319943 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.319906    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bjj8m"]
Sep 17 22:05:40.330609 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.330586    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-prmkq"]
Sep 17 22:05:40.334318 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.334297    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.336490 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.336467    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 22:05:40.336581 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.336519    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 22:05:40.336646 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.336587    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-5pvxl\""
Sep 17 22:05:40.341104 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.341076    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-prmkq"]
Sep 17 22:05:40.440224 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.440198    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8ef02a-a3ff-4091-bdf3-4b2605013977-metrics-certs\") pod \"odh-model-controller-7b966d7984-prmkq\" (UID: \"0e8ef02a-a3ff-4091-bdf3-4b2605013977\") " pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.440340 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.440229    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e8ef02a-a3ff-4091-bdf3-4b2605013977-cert\") pod \"odh-model-controller-7b966d7984-prmkq\" (UID: \"0e8ef02a-a3ff-4091-bdf3-4b2605013977\") " pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.440340 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.440312    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2beca00-626f-4e8a-849e-937f9a984152-tls-certs\") pod \"model-serving-api-7b954ffc95-bjj8m\" (UID: \"e2beca00-626f-4e8a-849e-937f9a984152\") " pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:40.440479 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.440358    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ljc\" (UniqueName: \"kubernetes.io/projected/0e8ef02a-a3ff-4091-bdf3-4b2605013977-kube-api-access-q5ljc\") pod \"odh-model-controller-7b966d7984-prmkq\" (UID: \"0e8ef02a-a3ff-4091-bdf3-4b2605013977\") " pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.440479 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.440389    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k642\" (UniqueName: \"kubernetes.io/projected/e2beca00-626f-4e8a-849e-937f9a984152-kube-api-access-2k642\") pod \"model-serving-api-7b954ffc95-bjj8m\" (UID: \"e2beca00-626f-4e8a-849e-937f9a984152\") " pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:40.540781 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.540751    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k642\" (UniqueName: \"kubernetes.io/projected/e2beca00-626f-4e8a-849e-937f9a984152-kube-api-access-2k642\") pod \"model-serving-api-7b954ffc95-bjj8m\" (UID: \"e2beca00-626f-4e8a-849e-937f9a984152\") " pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:40.540923 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.540793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8ef02a-a3ff-4091-bdf3-4b2605013977-metrics-certs\") pod \"odh-model-controller-7b966d7984-prmkq\" (UID: \"0e8ef02a-a3ff-4091-bdf3-4b2605013977\") " pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.540923 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.540819    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e8ef02a-a3ff-4091-bdf3-4b2605013977-cert\") pod \"odh-model-controller-7b966d7984-prmkq\" (UID: \"0e8ef02a-a3ff-4091-bdf3-4b2605013977\") " pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.540923 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.540878    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2beca00-626f-4e8a-849e-937f9a984152-tls-certs\") pod \"model-serving-api-7b954ffc95-bjj8m\" (UID: \"e2beca00-626f-4e8a-849e-937f9a984152\") " pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:40.541088 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.540924    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ljc\" (UniqueName: \"kubernetes.io/projected/0e8ef02a-a3ff-4091-bdf3-4b2605013977-kube-api-access-q5ljc\") pod \"odh-model-controller-7b966d7984-prmkq\" (UID: \"0e8ef02a-a3ff-4091-bdf3-4b2605013977\") " pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.541088 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:05:40.541054    2585 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 22:05:40.541187 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:05:40.541139    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2beca00-626f-4e8a-849e-937f9a984152-tls-certs podName:e2beca00-626f-4e8a-849e-937f9a984152 nodeName:}" failed. No retries permitted until 2026-09-17 22:05:41.041115868 +0000 UTC m=+442.903369338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e2beca00-626f-4e8a-849e-937f9a984152-tls-certs") pod "model-serving-api-7b954ffc95-bjj8m" (UID: "e2beca00-626f-4e8a-849e-937f9a984152") : secret "model-serving-api-tls" not found
Sep 17 22:05:40.543373 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.543352    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e8ef02a-a3ff-4091-bdf3-4b2605013977-cert\") pod \"odh-model-controller-7b966d7984-prmkq\" (UID: \"0e8ef02a-a3ff-4091-bdf3-4b2605013977\") " pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.543480 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.543400    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8ef02a-a3ff-4091-bdf3-4b2605013977-metrics-certs\") pod \"odh-model-controller-7b966d7984-prmkq\" (UID: \"0e8ef02a-a3ff-4091-bdf3-4b2605013977\") " pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.549290 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.549268    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k642\" (UniqueName: \"kubernetes.io/projected/e2beca00-626f-4e8a-849e-937f9a984152-kube-api-access-2k642\") pod \"model-serving-api-7b954ffc95-bjj8m\" (UID: \"e2beca00-626f-4e8a-849e-937f9a984152\") " pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:40.549497 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.549481    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ljc\" (UniqueName: \"kubernetes.io/projected/0e8ef02a-a3ff-4091-bdf3-4b2605013977-kube-api-access-q5ljc\") pod \"odh-model-controller-7b966d7984-prmkq\" (UID: \"0e8ef02a-a3ff-4091-bdf3-4b2605013977\") " pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.644609 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.644555    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:40.913815 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:40.913746    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-prmkq"]
Sep 17 22:05:40.916691 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:05:40.916657    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8ef02a_a3ff_4091_bdf3_4b2605013977.slice/crio-0fdfaed4089457d1ddda64f90ba46e35ef67ff8f1b69603fc7b5b90ead597f82 WatchSource:0}: Error finding container 0fdfaed4089457d1ddda64f90ba46e35ef67ff8f1b69603fc7b5b90ead597f82: Status 404 returned error can't find the container with id 0fdfaed4089457d1ddda64f90ba46e35ef67ff8f1b69603fc7b5b90ead597f82
Sep 17 22:05:41.045666 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:41.045635    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2beca00-626f-4e8a-849e-937f9a984152-tls-certs\") pod \"model-serving-api-7b954ffc95-bjj8m\" (UID: \"e2beca00-626f-4e8a-849e-937f9a984152\") " pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:41.048028 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:41.048011    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2beca00-626f-4e8a-849e-937f9a984152-tls-certs\") pod \"model-serving-api-7b954ffc95-bjj8m\" (UID: \"e2beca00-626f-4e8a-849e-937f9a984152\") " pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:41.162365 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:41.162337    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-prmkq" event={"ID":"0e8ef02a-a3ff-4091-bdf3-4b2605013977","Type":"ContainerStarted","Data":"0fdfaed4089457d1ddda64f90ba46e35ef67ff8f1b69603fc7b5b90ead597f82"}
Sep 17 22:05:41.223741 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:41.223686    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:41.494330 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:41.494297    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bjj8m"]
Sep 17 22:05:41.496903 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:05:41.496859    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2beca00_626f_4e8a_849e_937f9a984152.slice/crio-1ef774512d69329bbb5f214e800096e535e8ac68ab9cf74801ef47d1365495c3 WatchSource:0}: Error finding container 1ef774512d69329bbb5f214e800096e535e8ac68ab9cf74801ef47d1365495c3: Status 404 returned error can't find the container with id 1ef774512d69329bbb5f214e800096e535e8ac68ab9cf74801ef47d1365495c3
Sep 17 22:05:42.172750 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:42.172705    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bjj8m" event={"ID":"e2beca00-626f-4e8a-849e-937f9a984152","Type":"ContainerStarted","Data":"1ef774512d69329bbb5f214e800096e535e8ac68ab9cf74801ef47d1365495c3"}
Sep 17 22:05:45.192468 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:45.192404    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bjj8m" event={"ID":"e2beca00-626f-4e8a-849e-937f9a984152","Type":"ContainerStarted","Data":"df517bd4e1db1a7f44a07a4e2069992f05a491f6a96ff6c59f2e05b6dbd8d727"}
Sep 17 22:05:45.193019 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:45.192504    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:45.194174 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:45.194144    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-prmkq" event={"ID":"0e8ef02a-a3ff-4091-bdf3-4b2605013977","Type":"ContainerStarted","Data":"6d48d3e37059db0b40d5138458a355ac79a4567cd92eabbab561571af078241f"}
Sep 17 22:05:45.194325 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:45.194259    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:45.208995 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:45.208942    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-bjj8m" podStartSLOduration=2.070302556 podStartE2EDuration="5.208928049s" totalImagesPullingTime="3.138625493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:45.208050672 +0000 UTC m=+447.070304150" watchObservedRunningTime="2026-09-17 22:05:45.208928049 +0000 UTC m=+447.071181525"
Sep 17 22:05:56.201091 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:56.201062    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-prmkq"
Sep 17 22:05:56.202944 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:56.202920    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-bjj8m"
Sep 17 22:05:56.217599 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:05:56.217558    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-prmkq" podStartSLOduration=12.538204188 podStartE2EDuration="16.21754609s" totalImagesPullingTime="3.679341902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:45.222677981 +0000 UTC m=+447.084931458" watchObservedRunningTime="2026-09-17 22:05:56.21754609 +0000 UTC m=+458.079799565"
Sep 17 22:07:26.863329 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.863294    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"]
Sep 17 22:07:26.865682 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.865667    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:26.870038 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.870019    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:07:26.870976 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.870957    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:07:26.871184 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.870962    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 22:07:26.871265 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.871205    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:07:26.871265 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.871252    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:07:26.871376 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.871341    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 22:07:26.871493 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.871477    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:07:26.884700 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.884675    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"]
Sep 17 22:07:26.973040 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.973012    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad961151-5352-429d-a3c3-f5dd8f6106a1-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:26.973193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.973059    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjvwd\" (UniqueName: \"kubernetes.io/projected/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-api-access-pjvwd\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:26.973193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.973099    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:26.973193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.973125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad961151-5352-429d-a3c3-f5dd8f6106a1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:26.973193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:26.973142    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.074447 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.074405    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad961151-5352-429d-a3c3-f5dd8f6106a1-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.074571 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.074493    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjvwd\" (UniqueName: \"kubernetes.io/projected/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-api-access-pjvwd\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.074571 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.074532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.074571 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:07:27.074564    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 22:07:27.074690 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.074570    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad961151-5352-429d-a3c3-f5dd8f6106a1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.074690 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.074596    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.074690 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:07:27.074626    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad961151-5352-429d-a3c3-f5dd8f6106a1-proxy-tls podName:ad961151-5352-429d-a3c3-f5dd8f6106a1 nodeName:}" failed. No retries permitted until 2026-09-17 22:07:27.574605423 +0000 UTC m=+549.436858879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad961151-5352-429d-a3c3-f5dd8f6106a1-proxy-tls") pod "isvc-canary-rollback-predictor-76cbc5464-bxt5z" (UID: "ad961151-5352-429d-a3c3-f5dd8f6106a1") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 22:07:27.075033 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.075010    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad961151-5352-429d-a3c3-f5dd8f6106a1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.075209 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.075192    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.075246 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.075211    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.089290 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.089261    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjvwd\" (UniqueName: \"kubernetes.io/projected/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-api-access-pjvwd\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.580063 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.580032    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad961151-5352-429d-a3c3-f5dd8f6106a1-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.582466 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.582416    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad961151-5352-429d-a3c3-f5dd8f6106a1-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-bxt5z\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:27.776056 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:27.776025    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:28.049221 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:28.049195    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"]
Sep 17 22:07:28.051924 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:07:28.051896    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad961151_5352_429d_a3c3_f5dd8f6106a1.slice/crio-638eb55841ae19e3bbdb545ecbef813679318baee60e5b881237706f5a07fc95 WatchSource:0}: Error finding container 638eb55841ae19e3bbdb545ecbef813679318baee60e5b881237706f5a07fc95: Status 404 returned error can't find the container with id 638eb55841ae19e3bbdb545ecbef813679318baee60e5b881237706f5a07fc95
Sep 17 22:07:28.575396 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:28.575357    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" event={"ID":"ad961151-5352-429d-a3c3-f5dd8f6106a1","Type":"ContainerStarted","Data":"638eb55841ae19e3bbdb545ecbef813679318baee60e5b881237706f5a07fc95"}
Sep 17 22:07:31.590046 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:31.590007    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" event={"ID":"ad961151-5352-429d-a3c3-f5dd8f6106a1","Type":"ContainerStarted","Data":"07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe"}
Sep 17 22:07:35.606548 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:35.606508    2585 generic.go:356] "Generic (PLEG): container finished" podID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerID="07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe" exitCode=0
Sep 17 22:07:35.606963 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:35.606577    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" event={"ID":"ad961151-5352-429d-a3c3-f5dd8f6106a1","Type":"ContainerDied","Data":"07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe"}
Sep 17 22:07:49.685861 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:49.685820    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" event={"ID":"ad961151-5352-429d-a3c3-f5dd8f6106a1","Type":"ContainerStarted","Data":"359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3"}
Sep 17 22:07:52.701709 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:52.701666    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" event={"ID":"ad961151-5352-429d-a3c3-f5dd8f6106a1","Type":"ContainerStarted","Data":"d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904"}
Sep 17 22:07:52.702103 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:52.701839    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:52.720083 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:52.720037    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podStartSLOduration=3.008889031 podStartE2EDuration="26.720020828s" totalImagesPullingTime="19.694201705s" totalInitContainerRuntime="4.016930092s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:07:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:52.718409809 +0000 UTC m=+574.580663323" watchObservedRunningTime="2026-09-17 22:07:52.720020828 +0000 UTC m=+574.582274355"
Sep 17 22:07:53.706363 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:53.706325    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:53.707720 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:53.707690    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 22:07:54.709444 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:54.709387    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 22:07:59.713846 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:59.713818    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:07:59.714330 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:07:59.714300    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 22:08:09.714549 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:08:09.714509    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 22:08:19.714315 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:08:19.714271    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 22:08:20.380274 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:08:20.380250    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:08:20.381138 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:08:20.381108    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:08:29.714875 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:08:29.714838    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 22:08:39.714848 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:08:39.714804    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 22:08:49.714578 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:08:49.714538    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 22:08:59.715731 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:08:59.715696    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:09:21.819024 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.818992    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"]
Sep 17 22:09:21.821336 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.821319    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:21.823783 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.823761    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-49e3c-kube-rbac-proxy-sar-config\""
Sep 17 22:09:21.823900 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.823795    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-49e3c-predictor-serving-cert\""
Sep 17 22:09:21.832129 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.832106    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"]
Sep 17 22:09:21.922559 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.922530    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-cabundle-cert\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:21.922732 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.922569    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e727a2e-19a7-4ba6-ade7-785008b73897-proxy-tls\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:21.922732 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.922592    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzt5s\" (UniqueName: \"kubernetes.io/projected/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-api-access-gzt5s\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:21.922732 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.922656    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e727a2e-19a7-4ba6-ade7-785008b73897-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:21.922732 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.922686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:21.963595 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.963559    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"]
Sep 17 22:09:21.966518 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.966499    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:21.968728 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.968704    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-49e3c-kube-rbac-proxy-sar-config\""
Sep 17 22:09:21.968830 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.968748    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-49e3c-predictor-serving-cert\""
Sep 17 22:09:21.978399 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:21.978370    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"]
Sep 17 22:09:22.023967 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.023942    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e727a2e-19a7-4ba6-ade7-785008b73897-proxy-tls\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.024069 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.023971    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzt5s\" (UniqueName: \"kubernetes.io/projected/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-api-access-gzt5s\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.024069 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.024006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e727a2e-19a7-4ba6-ade7-785008b73897-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.024069 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.024024    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.024232 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:09:22.024114    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-serving-cert: secret "isvc-sklearn-graph-raw-49e3c-predictor-serving-cert" not found
Sep 17 22:09:22.024232 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.024195    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-cabundle-cert\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.024232 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:09:22.024202    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e727a2e-19a7-4ba6-ade7-785008b73897-proxy-tls podName:6e727a2e-19a7-4ba6-ade7-785008b73897 nodeName:}" failed. No retries permitted until 2026-09-17 22:09:22.52418246 +0000 UTC m=+664.386435918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e727a2e-19a7-4ba6-ade7-785008b73897-proxy-tls") pod "isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" (UID: "6e727a2e-19a7-4ba6-ade7-785008b73897") : secret "isvc-sklearn-graph-raw-49e3c-predictor-serving-cert" not found
Sep 17 22:09:22.024392 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.024358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e727a2e-19a7-4ba6-ade7-785008b73897-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.024680 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.024663    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.024762 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.024747    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-cabundle-cert\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.035959 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.035940    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzt5s\" (UniqueName: \"kubernetes.io/projected/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-api-access-gzt5s\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.125106 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.125043    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccb127e-b601-466e-bf4e-6f24ce34060d-proxy-tls\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.125211 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.125098    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.125211 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.125184    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.125326 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.125216    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfjrl\" (UniqueName: \"kubernetes.io/projected/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-api-access-pfjrl\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.125326 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.125256    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccb127e-b601-466e-bf4e-6f24ce34060d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.225531 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.225507    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfjrl\" (UniqueName: \"kubernetes.io/projected/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-api-access-pfjrl\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.225635 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.225541    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccb127e-b601-466e-bf4e-6f24ce34060d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.225635 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.225567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccb127e-b601-466e-bf4e-6f24ce34060d-proxy-tls\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.225738 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.225690    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.225793 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.225765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.225942 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.225923    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccb127e-b601-466e-bf4e-6f24ce34060d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.226294 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.226276    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.226335 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.226296    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.228004 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.227986    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccb127e-b601-466e-bf4e-6f24ce34060d-proxy-tls\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.233581 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.233564    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfjrl\" (UniqueName: \"kubernetes.io/projected/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-api-access-pfjrl\") pod \"isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.276533 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.276511    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:22.529350 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.529323    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e727a2e-19a7-4ba6-ade7-785008b73897-proxy-tls\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.531865 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.531846    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e727a2e-19a7-4ba6-ade7-785008b73897-proxy-tls\") pod \"isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:22.546093 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.546026    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"]
Sep 17 22:09:22.548790 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:09:22.548764    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccb127e_b601_466e_bf4e_6f24ce34060d.slice/crio-2163ac874626d3052f0698d8c0c9ab16f2f62473628c05e2a7afe385c0cf7ea0 WatchSource:0}: Error finding container 2163ac874626d3052f0698d8c0c9ab16f2f62473628c05e2a7afe385c0cf7ea0: Status 404 returned error can't find the container with id 2163ac874626d3052f0698d8c0c9ab16f2f62473628c05e2a7afe385c0cf7ea0
Sep 17 22:09:22.733202 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:22.733163    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:23.014277 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:23.014246    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"]
Sep 17 22:09:23.017062 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:09:23.017027    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e727a2e_19a7_4ba6_ade7_785008b73897.slice/crio-6afeb1b1b22951fbcfe100e9d6d04035d0f67115ee680f5f13b8941da62ee3b2 WatchSource:0}: Error finding container 6afeb1b1b22951fbcfe100e9d6d04035d0f67115ee680f5f13b8941da62ee3b2: Status 404 returned error can't find the container with id 6afeb1b1b22951fbcfe100e9d6d04035d0f67115ee680f5f13b8941da62ee3b2
Sep 17 22:09:23.051036 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:23.050956    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" event={"ID":"6e727a2e-19a7-4ba6-ade7-785008b73897","Type":"ContainerStarted","Data":"6afeb1b1b22951fbcfe100e9d6d04035d0f67115ee680f5f13b8941da62ee3b2"}
Sep 17 22:09:23.052530 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:23.052498    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" event={"ID":"8ccb127e-b601-466e-bf4e-6f24ce34060d","Type":"ContainerStarted","Data":"e82aaef40ffe97e9f8e1d7201d0853c4cb6be830ceb919be14babead0ed39c3f"}
Sep 17 22:09:23.052617 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:23.052551    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" event={"ID":"8ccb127e-b601-466e-bf4e-6f24ce34060d","Type":"ContainerStarted","Data":"2163ac874626d3052f0698d8c0c9ab16f2f62473628c05e2a7afe385c0cf7ea0"}
Sep 17 22:09:24.057628 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:24.057592    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" event={"ID":"6e727a2e-19a7-4ba6-ade7-785008b73897","Type":"ContainerStarted","Data":"1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f"}
Sep 17 22:09:27.071811 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:27.071776    2585 generic.go:356] "Generic (PLEG): container finished" podID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerID="e82aaef40ffe97e9f8e1d7201d0853c4cb6be830ceb919be14babead0ed39c3f" exitCode=0
Sep 17 22:09:27.072222 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:27.071859    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" event={"ID":"8ccb127e-b601-466e-bf4e-6f24ce34060d","Type":"ContainerDied","Data":"e82aaef40ffe97e9f8e1d7201d0853c4cb6be830ceb919be14babead0ed39c3f"}
Sep 17 22:09:27.073392 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:27.073370    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:09:27.073717 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:27.073693    2585 generic.go:356] "Generic (PLEG): container finished" podID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerID="1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f" exitCode=0
Sep 17 22:09:27.073817 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:27.073718    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" event={"ID":"6e727a2e-19a7-4ba6-ade7-785008b73897","Type":"ContainerDied","Data":"1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f"}
Sep 17 22:09:28.080908 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:28.080874    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" event={"ID":"6e727a2e-19a7-4ba6-ade7-785008b73897","Type":"ContainerStarted","Data":"8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151"}
Sep 17 22:09:28.081305 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:28.080940    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" event={"ID":"6e727a2e-19a7-4ba6-ade7-785008b73897","Type":"ContainerStarted","Data":"04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb"}
Sep 17 22:09:28.081305 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:28.081131    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:28.100322 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:28.100251    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podStartSLOduration=3.476743947 podStartE2EDuration="7.100232904s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.623488957s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:09:28.09910291 +0000 UTC m=+669.961356414" watchObservedRunningTime="2026-09-17 22:09:28.100232904 +0000 UTC m=+669.962486382"
Sep 17 22:09:29.087718 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:29.087635    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:29.089189 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:29.089152    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:30.093462 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:30.093405    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:35.099333 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:35.099300    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:09:35.099999 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:35.099965    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:45.100797 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:45.100743    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:09:47.174220 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:47.174185    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" event={"ID":"8ccb127e-b601-466e-bf4e-6f24ce34060d","Type":"ContainerStarted","Data":"e6283a825abc8f4c150640b2acd60c10013657cdc91b820bc56561f23c1db511"}
Sep 17 22:09:47.174628 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:47.174248    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" event={"ID":"8ccb127e-b601-466e-bf4e-6f24ce34060d","Type":"ContainerStarted","Data":"5cd4f10a8c56dac6ccbd9b7ef0437adbe5db744abd0b5d7ccdde31096c87921c"}
Sep 17 22:09:47.174628 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:47.174453    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:47.198581 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:47.198523    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podStartSLOduration=3.089382688 podStartE2EDuration="26.198506814s" totalImagesPullingTime="19.201009704s" totalInitContainerRuntime="3.908114422s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:09:47.197455864 +0000 UTC m=+689.059709332" watchObservedRunningTime="2026-09-17 22:09:47.198506814 +0000 UTC m=+689.060760290"
Sep 17 22:09:48.184250 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:48.182803    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:48.184250 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:48.182901    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:09:49.185065 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:49.185027    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:09:54.191637 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:54.191601    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:09:54.192377 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:54.192345    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:09:55.100710 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:09:55.100669    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:04.193299 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:04.193195    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:10:05.100060 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:05.100015    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:14.192787 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:14.192732    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:10:15.100897 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:15.100843    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:24.192232 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:24.192183    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:10:25.100869 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:25.100824    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:34.193308 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:34.193257    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:10:35.100700 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:35.100658    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:44.193235 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:44.193191    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:10:45.100102 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:45.100059    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 22:10:46.787383 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:46.787339    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"]
Sep 17 22:10:46.788038 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:46.787712    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" containerID="cri-o://359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3" gracePeriod=30
Sep 17 22:10:46.788161 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:46.788126    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kube-rbac-proxy" containerID="cri-o://d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904" gracePeriod=30
Sep 17 22:10:47.431865 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:47.431827    2585 generic.go:356] "Generic (PLEG): container finished" podID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerID="d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904" exitCode=2
Sep 17 22:10:47.432041 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:47.431900    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" event={"ID":"ad961151-5352-429d-a3c3-f5dd8f6106a1","Type":"ContainerDied","Data":"d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904"}
Sep 17 22:10:49.710398 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:49.710351    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 22:10:49.714738 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:49.714707    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 22:10:51.459571 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:51.459538    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:10:52.056356 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.056328    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:10:52.104185 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.104149    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad961151-5352-429d-a3c3-f5dd8f6106a1-kserve-provision-location\") pod \"ad961151-5352-429d-a3c3-f5dd8f6106a1\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") "
Sep 17 22:10:52.104364 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.104225    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad961151-5352-429d-a3c3-f5dd8f6106a1-proxy-tls\") pod \"ad961151-5352-429d-a3c3-f5dd8f6106a1\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") "
Sep 17 22:10:52.104364 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.104249    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-rbac-proxy-sar-config\") pod \"ad961151-5352-429d-a3c3-f5dd8f6106a1\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") "
Sep 17 22:10:52.104364 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.104286    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-cabundle-cert\") pod \"ad961151-5352-429d-a3c3-f5dd8f6106a1\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") "
Sep 17 22:10:52.104364 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.104304    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjvwd\" (UniqueName: \"kubernetes.io/projected/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-api-access-pjvwd\") pod \"ad961151-5352-429d-a3c3-f5dd8f6106a1\" (UID: \"ad961151-5352-429d-a3c3-f5dd8f6106a1\") "
Sep 17 22:10:52.104644 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.104550    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad961151-5352-429d-a3c3-f5dd8f6106a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad961151-5352-429d-a3c3-f5dd8f6106a1" (UID: "ad961151-5352-429d-a3c3-f5dd8f6106a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:10:52.104699 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.104679    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad961151-5352-429d-a3c3-f5dd8f6106a1" (UID: "ad961151-5352-429d-a3c3-f5dd8f6106a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:52.104755 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.104704    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad961151-5352-429d-a3c3-f5dd8f6106a1" (UID: "ad961151-5352-429d-a3c3-f5dd8f6106a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:10:52.106878 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.106850    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-api-access-pjvwd" (OuterVolumeSpecName: "kube-api-access-pjvwd") pod "ad961151-5352-429d-a3c3-f5dd8f6106a1" (UID: "ad961151-5352-429d-a3c3-f5dd8f6106a1"). InnerVolumeSpecName "kube-api-access-pjvwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:10:52.107020 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.106916    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad961151-5352-429d-a3c3-f5dd8f6106a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad961151-5352-429d-a3c3-f5dd8f6106a1" (UID: "ad961151-5352-429d-a3c3-f5dd8f6106a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:10:52.189718 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.189622    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"]
Sep 17 22:10:52.190016 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.189998    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kube-rbac-proxy"
Sep 17 22:10:52.190103 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.190020    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kube-rbac-proxy"
Sep 17 22:10:52.190103 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.190035    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container"
Sep 17 22:10:52.190103 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.190043    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container"
Sep 17 22:10:52.190103 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.190076    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="storage-initializer"
Sep 17 22:10:52.190103 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.190085    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="storage-initializer"
Sep 17 22:10:52.190341 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.190166    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kube-rbac-proxy"
Sep 17 22:10:52.190341 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.190179    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerName="kserve-container"
Sep 17 22:10:52.192711 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.192689    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.195209 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.195181    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 22:10:52.195385 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.195229    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 22:10:52.205200 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.205170    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"]
Sep 17 22:10:52.205480 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.205462    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-cabundle-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:10:52.205562 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.205482    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjvwd\" (UniqueName: \"kubernetes.io/projected/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-api-access-pjvwd\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:10:52.205562 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.205492    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad961151-5352-429d-a3c3-f5dd8f6106a1-kserve-provision-location\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:10:52.205562 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.205502    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad961151-5352-429d-a3c3-f5dd8f6106a1-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:10:52.205562 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.205510    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad961151-5352-429d-a3c3-f5dd8f6106a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:10:52.306476 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.306411    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9243e38-c20b-4754-99a2-5b73d13420f5-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.306650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.306536    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9243e38-c20b-4754-99a2-5b73d13420f5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.306650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.306567    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.306650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.306591    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhf8w\" (UniqueName: \"kubernetes.io/projected/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-api-access-nhf8w\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.306650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.306608    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.407041 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.406998    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.407226 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.407082    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9243e38-c20b-4754-99a2-5b73d13420f5-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.407226 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.407138    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9243e38-c20b-4754-99a2-5b73d13420f5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.407226 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.407162    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.407226 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.407193    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhf8w\" (UniqueName: \"kubernetes.io/projected/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-api-access-nhf8w\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.407378 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:10:52.407249    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 22:10:52.407378 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:10:52.407337    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9243e38-c20b-4754-99a2-5b73d13420f5-proxy-tls podName:b9243e38-c20b-4754-99a2-5b73d13420f5 nodeName:}" failed. No retries permitted until 2026-09-17 22:10:52.907313667 +0000 UTC m=+754.769567121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b9243e38-c20b-4754-99a2-5b73d13420f5-proxy-tls") pod "isvc-canary-stop-predictor-646f5bdbcb-dgzkq" (UID: "b9243e38-c20b-4754-99a2-5b73d13420f5") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 22:10:52.407726 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.407704    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9243e38-c20b-4754-99a2-5b73d13420f5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.407962 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.407944    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.407962 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.407955    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-cabundle-cert\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.415731 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.415698    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhf8w\" (UniqueName: \"kubernetes.io/projected/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-api-access-nhf8w\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.454690 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.454601    2585 generic.go:356] "Generic (PLEG): container finished" podID="ad961151-5352-429d-a3c3-f5dd8f6106a1" containerID="359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3" exitCode=0
Sep 17 22:10:52.454690 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.454665    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" event={"ID":"ad961151-5352-429d-a3c3-f5dd8f6106a1","Type":"ContainerDied","Data":"359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3"}
Sep 17 22:10:52.454903 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.454706    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"
Sep 17 22:10:52.454903 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.454723    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z" event={"ID":"ad961151-5352-429d-a3c3-f5dd8f6106a1","Type":"ContainerDied","Data":"638eb55841ae19e3bbdb545ecbef813679318baee60e5b881237706f5a07fc95"}
Sep 17 22:10:52.454903 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.454752    2585 scope.go:117] "RemoveContainer" containerID="d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904"
Sep 17 22:10:52.464055 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.464037    2585 scope.go:117] "RemoveContainer" containerID="359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3"
Sep 17 22:10:52.473218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.473196    2585 scope.go:117] "RemoveContainer" containerID="07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe"
Sep 17 22:10:52.479152 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.479122    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"]
Sep 17 22:10:52.482414 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.482391    2585 scope.go:117] "RemoveContainer" containerID="d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904"
Sep 17 22:10:52.482814 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:10:52.482790    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904\": container with ID starting with d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904 not found: ID does not exist" containerID="d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904"
Sep 17 22:10:52.482889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.482825    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904"} err="failed to get container status \"d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904\": rpc error: code = NotFound desc = could not find container \"d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904\": container with ID starting with d1ab061ffd565aff1ee636cc4428b2409888ea6d94f884301d01a03d6516f904 not found: ID does not exist"
Sep 17 22:10:52.482889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.482844    2585 scope.go:117] "RemoveContainer" containerID="359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3"
Sep 17 22:10:52.483164 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:10:52.483141    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3\": container with ID starting with 359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3 not found: ID does not exist" containerID="359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3"
Sep 17 22:10:52.483207 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.483175    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3"} err="failed to get container status \"359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3\": rpc error: code = NotFound desc = could not find container \"359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3\": container with ID starting with 359e6b01020b948958df7dacc9d7531f1046896a86736708261cd021106f15a3 not found: ID does not exist"
Sep 17 22:10:52.483207 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.483199    2585 scope.go:117] "RemoveContainer" containerID="07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe"
Sep 17 22:10:52.483479 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:10:52.483462    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe\": container with ID starting with 07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe not found: ID does not exist" containerID="07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe"
Sep 17 22:10:52.483563 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.483484    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe"} err="failed to get container status \"07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe\": rpc error: code = NotFound desc = could not find container \"07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe\": container with ID starting with 07162a7d9d3a6fae002c0790feda112193f0641fa51d42a71f0e6287c5433ffe not found: ID does not exist"
Sep 17 22:10:52.483693 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.483675    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-bxt5z"]
Sep 17 22:10:52.911393 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.911344    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9243e38-c20b-4754-99a2-5b73d13420f5-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:52.914039 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:52.914018    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9243e38-c20b-4754-99a2-5b73d13420f5-proxy-tls\") pod \"isvc-canary-stop-predictor-646f5bdbcb-dgzkq\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:53.102315 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:53.102276    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:53.377295 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:53.377260    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"]
Sep 17 22:10:53.380456 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:10:53.380407    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9243e38_c20b_4754_99a2_5b73d13420f5.slice/crio-561d570a6726101de64d7c7effa612bdd3290a555ec6a2013312219470a68baf WatchSource:0}: Error finding container 561d570a6726101de64d7c7effa612bdd3290a555ec6a2013312219470a68baf: Status 404 returned error can't find the container with id 561d570a6726101de64d7c7effa612bdd3290a555ec6a2013312219470a68baf
Sep 17 22:10:53.461482 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:53.461450    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" event={"ID":"b9243e38-c20b-4754-99a2-5b73d13420f5","Type":"ContainerStarted","Data":"89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b"}
Sep 17 22:10:53.461659 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:53.461512    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" event={"ID":"b9243e38-c20b-4754-99a2-5b73d13420f5","Type":"ContainerStarted","Data":"561d570a6726101de64d7c7effa612bdd3290a555ec6a2013312219470a68baf"}
Sep 17 22:10:54.192560 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:54.192512    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:10:54.464408 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:54.464306    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad961151-5352-429d-a3c3-f5dd8f6106a1" path="/var/lib/kubelet/pods/ad961151-5352-429d-a3c3-f5dd8f6106a1/volumes"
Sep 17 22:10:56.464632 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:56.464599    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:10:57.479042 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:57.479001    2585 generic.go:356] "Generic (PLEG): container finished" podID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerID="89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b" exitCode=0
Sep 17 22:10:57.479415 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:57.479076    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" event={"ID":"b9243e38-c20b-4754-99a2-5b73d13420f5","Type":"ContainerDied","Data":"89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b"}
Sep 17 22:10:58.486400 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:58.486358    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" event={"ID":"b9243e38-c20b-4754-99a2-5b73d13420f5","Type":"ContainerStarted","Data":"77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65"}
Sep 17 22:10:58.486800 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:58.486452    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" event={"ID":"b9243e38-c20b-4754-99a2-5b73d13420f5","Type":"ContainerStarted","Data":"39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2"}
Sep 17 22:10:58.486800 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:58.486655    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:58.506860 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:58.506800    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podStartSLOduration=2.548985431 podStartE2EDuration="6.50677451s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.957789079s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:10:58.504780895 +0000 UTC m=+760.367034372" watchObservedRunningTime="2026-09-17 22:10:58.50677451 +0000 UTC m=+760.369027987"
Sep 17 22:10:59.491302 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:59.491268    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:10:59.492675 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:10:59.492645    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 22:11:00.495193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:00.495153    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 22:11:05.499818 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:05.499791    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:11:05.500394 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:05.500364    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 22:11:12.040242 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.040211    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"]
Sep 17 22:11:12.040665 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.040514    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container" containerID="cri-o://04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb" gracePeriod=30
Sep 17 22:11:12.040665 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.040555    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kube-rbac-proxy" containerID="cri-o://8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151" gracePeriod=30
Sep 17 22:11:12.083503 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.083464    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"]
Sep 17 22:11:12.083801 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.083765    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" containerID="cri-o://5cd4f10a8c56dac6ccbd9b7ef0437adbe5db744abd0b5d7ccdde31096c87921c" gracePeriod=30
Sep 17 22:11:12.083968 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.083822    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kube-rbac-proxy" containerID="cri-o://e6283a825abc8f4c150640b2acd60c10013657cdc91b820bc56561f23c1db511" gracePeriod=30
Sep 17 22:11:12.554171 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.554133    2585 generic.go:356] "Generic (PLEG): container finished" podID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerID="e6283a825abc8f4c150640b2acd60c10013657cdc91b820bc56561f23c1db511" exitCode=2
Sep 17 22:11:12.554364 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.554206    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" event={"ID":"8ccb127e-b601-466e-bf4e-6f24ce34060d","Type":"ContainerDied","Data":"e6283a825abc8f4c150640b2acd60c10013657cdc91b820bc56561f23c1db511"}
Sep 17 22:11:12.556380 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.556356    2585 generic.go:356] "Generic (PLEG): container finished" podID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerID="8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151" exitCode=2
Sep 17 22:11:12.556525 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:12.556414    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" event={"ID":"6e727a2e-19a7-4ba6-ade7-785008b73897","Type":"ContainerDied","Data":"8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151"}
Sep 17 22:11:14.185386 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:14.185346    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 22:11:15.094693 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:15.094641    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 22:11:15.501317 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:15.501232    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 22:11:16.459504 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.459462    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 22:11:16.576905 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.576875    2585 generic.go:356] "Generic (PLEG): container finished" podID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerID="5cd4f10a8c56dac6ccbd9b7ef0437adbe5db744abd0b5d7ccdde31096c87921c" exitCode=0
Sep 17 22:11:16.577313 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.576947    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" event={"ID":"8ccb127e-b601-466e-bf4e-6f24ce34060d","Type":"ContainerDied","Data":"5cd4f10a8c56dac6ccbd9b7ef0437adbe5db744abd0b5d7ccdde31096c87921c"}
Sep 17 22:11:16.628826 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.628791    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:11:16.709516 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.709390    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfjrl\" (UniqueName: \"kubernetes.io/projected/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-api-access-pfjrl\") pod \"8ccb127e-b601-466e-bf4e-6f24ce34060d\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") "
Sep 17 22:11:16.709516 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.709471    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccb127e-b601-466e-bf4e-6f24ce34060d-proxy-tls\") pod \"8ccb127e-b601-466e-bf4e-6f24ce34060d\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") "
Sep 17 22:11:16.709516 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.709503    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-rbac-proxy-sar-config\") pod \"8ccb127e-b601-466e-bf4e-6f24ce34060d\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") "
Sep 17 22:11:16.709807 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.709555    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-cabundle-cert\") pod \"8ccb127e-b601-466e-bf4e-6f24ce34060d\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") "
Sep 17 22:11:16.709807 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.709581    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccb127e-b601-466e-bf4e-6f24ce34060d-kserve-provision-location\") pod \"8ccb127e-b601-466e-bf4e-6f24ce34060d\" (UID: \"8ccb127e-b601-466e-bf4e-6f24ce34060d\") "
Sep 17 22:11:16.709924 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.709899    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ccb127e-b601-466e-bf4e-6f24ce34060d" (UID: "8ccb127e-b601-466e-bf4e-6f24ce34060d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:11:16.709971 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.709942    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ccb127e-b601-466e-bf4e-6f24ce34060d" (UID: "8ccb127e-b601-466e-bf4e-6f24ce34060d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:11:16.709971 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.709951    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccb127e-b601-466e-bf4e-6f24ce34060d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ccb127e-b601-466e-bf4e-6f24ce34060d" (UID: "8ccb127e-b601-466e-bf4e-6f24ce34060d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:11:16.711912 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.711874    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ccb127e-b601-466e-bf4e-6f24ce34060d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ccb127e-b601-466e-bf4e-6f24ce34060d" (UID: "8ccb127e-b601-466e-bf4e-6f24ce34060d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:11:16.712032 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.711957    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-api-access-pfjrl" (OuterVolumeSpecName: "kube-api-access-pfjrl") pod "8ccb127e-b601-466e-bf4e-6f24ce34060d" (UID: "8ccb127e-b601-466e-bf4e-6f24ce34060d"). InnerVolumeSpecName "kube-api-access-pfjrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:11:16.811149 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.811111    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfjrl\" (UniqueName: \"kubernetes.io/projected/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-api-access-pfjrl\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:16.811149 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.811142    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ccb127e-b601-466e-bf4e-6f24ce34060d-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:16.811149 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.811154    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:16.811382 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.811163    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ccb127e-b601-466e-bf4e-6f24ce34060d-cabundle-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:16.811382 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:16.811181    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ccb127e-b601-466e-bf4e-6f24ce34060d-kserve-provision-location\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:17.396777 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.396737    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:11:17.515943 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.515831    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzt5s\" (UniqueName: \"kubernetes.io/projected/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-api-access-gzt5s\") pod \"6e727a2e-19a7-4ba6-ade7-785008b73897\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") "
Sep 17 22:11:17.515943 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.515897    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e727a2e-19a7-4ba6-ade7-785008b73897-proxy-tls\") pod \"6e727a2e-19a7-4ba6-ade7-785008b73897\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") "
Sep 17 22:11:17.515943 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.515948    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e727a2e-19a7-4ba6-ade7-785008b73897-kserve-provision-location\") pod \"6e727a2e-19a7-4ba6-ade7-785008b73897\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") "
Sep 17 22:11:17.516172 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.515970    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-rbac-proxy-sar-config\") pod \"6e727a2e-19a7-4ba6-ade7-785008b73897\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") "
Sep 17 22:11:17.516172 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.516017    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-cabundle-cert\") pod \"6e727a2e-19a7-4ba6-ade7-785008b73897\" (UID: \"6e727a2e-19a7-4ba6-ade7-785008b73897\") "
Sep 17 22:11:17.516390 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.516291    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e727a2e-19a7-4ba6-ade7-785008b73897-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e727a2e-19a7-4ba6-ade7-785008b73897" (UID: "6e727a2e-19a7-4ba6-ade7-785008b73897"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:11:17.516390 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.516375    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e727a2e-19a7-4ba6-ade7-785008b73897" (UID: "6e727a2e-19a7-4ba6-ade7-785008b73897"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:11:17.516539 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.516514    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e727a2e-19a7-4ba6-ade7-785008b73897" (UID: "6e727a2e-19a7-4ba6-ade7-785008b73897"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:11:17.518286 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.518259    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e727a2e-19a7-4ba6-ade7-785008b73897-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e727a2e-19a7-4ba6-ade7-785008b73897" (UID: "6e727a2e-19a7-4ba6-ade7-785008b73897"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:11:17.518286 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.518269    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-api-access-gzt5s" (OuterVolumeSpecName: "kube-api-access-gzt5s") pod "6e727a2e-19a7-4ba6-ade7-785008b73897" (UID: "6e727a2e-19a7-4ba6-ade7-785008b73897"). InnerVolumeSpecName "kube-api-access-gzt5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:11:17.583123 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.583087    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"
Sep 17 22:11:17.583636 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.583091    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf" event={"ID":"8ccb127e-b601-466e-bf4e-6f24ce34060d","Type":"ContainerDied","Data":"2163ac874626d3052f0698d8c0c9ab16f2f62473628c05e2a7afe385c0cf7ea0"}
Sep 17 22:11:17.583636 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.583248    2585 scope.go:117] "RemoveContainer" containerID="e6283a825abc8f4c150640b2acd60c10013657cdc91b820bc56561f23c1db511"
Sep 17 22:11:17.585735 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.585709    2585 generic.go:356] "Generic (PLEG): container finished" podID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerID="04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb" exitCode=0
Sep 17 22:11:17.585852 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.585764    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" event={"ID":"6e727a2e-19a7-4ba6-ade7-785008b73897","Type":"ContainerDied","Data":"04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb"}
Sep 17 22:11:17.585852 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.585812    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc" event={"ID":"6e727a2e-19a7-4ba6-ade7-785008b73897","Type":"ContainerDied","Data":"6afeb1b1b22951fbcfe100e9d6d04035d0f67115ee680f5f13b8941da62ee3b2"}
Sep 17 22:11:17.585852 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.585820    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"
Sep 17 22:11:17.593574 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.593523    2585 scope.go:117] "RemoveContainer" containerID="5cd4f10a8c56dac6ccbd9b7ef0437adbe5db744abd0b5d7ccdde31096c87921c"
Sep 17 22:11:17.602894 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.602873    2585 scope.go:117] "RemoveContainer" containerID="e82aaef40ffe97e9f8e1d7201d0853c4cb6be830ceb919be14babead0ed39c3f"
Sep 17 22:11:17.607565 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.607533    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"]
Sep 17 22:11:17.611718 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.611686    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-49e3c-predictor-d6bb58445-vbxqf"]
Sep 17 22:11:17.613318 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.613296    2585 scope.go:117] "RemoveContainer" containerID="8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151"
Sep 17 22:11:17.616901 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.616875    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-cabundle-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:17.616901 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.616898    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzt5s\" (UniqueName: \"kubernetes.io/projected/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-api-access-gzt5s\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:17.616901 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.616908    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e727a2e-19a7-4ba6-ade7-785008b73897-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:17.617146 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.616919    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e727a2e-19a7-4ba6-ade7-785008b73897-kserve-provision-location\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:17.617146 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.616927    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e727a2e-19a7-4ba6-ade7-785008b73897-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:11:17.622217 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.622193    2585 scope.go:117] "RemoveContainer" containerID="04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb"
Sep 17 22:11:17.622762 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.622736    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"]
Sep 17 22:11:17.628290 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.628253    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-49e3c-predictor-b6f4cf85-dm4nc"]
Sep 17 22:11:17.631583 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.631563    2585 scope.go:117] "RemoveContainer" containerID="1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f"
Sep 17 22:11:17.640187 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.640166    2585 scope.go:117] "RemoveContainer" containerID="8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151"
Sep 17 22:11:17.640558 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:11:17.640540    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151\": container with ID starting with 8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151 not found: ID does not exist" containerID="8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151"
Sep 17 22:11:17.640617 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.640568    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151"} err="failed to get container status \"8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151\": rpc error: code = NotFound desc = could not find container \"8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151\": container with ID starting with 8ae2a42b5cf74fea0b308b012c0a2bf322e55bbed997ac8d15a8f4df1d39c151 not found: ID does not exist"
Sep 17 22:11:17.640617 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.640588    2585 scope.go:117] "RemoveContainer" containerID="04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb"
Sep 17 22:11:17.640881 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:11:17.640866    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb\": container with ID starting with 04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb not found: ID does not exist" containerID="04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb"
Sep 17 22:11:17.640937 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.640887    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb"} err="failed to get container status \"04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb\": rpc error: code = NotFound desc = could not find container \"04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb\": container with ID starting with 04b5bc2547e1ddc46cc52496a68538f6b7463038ce96254e765ed53ac9472beb not found: ID does not exist"
Sep 17 22:11:17.640937 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.640902    2585 scope.go:117] "RemoveContainer" containerID="1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f"
Sep 17 22:11:17.641168 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:11:17.641147    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f\": container with ID starting with 1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f not found: ID does not exist" containerID="1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f"
Sep 17 22:11:17.641204 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:17.641178    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f"} err="failed to get container status \"1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f\": rpc error: code = NotFound desc = could not find container \"1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f\": container with ID starting with 1567a37a82c22d2c15aeed9dec1172d581692835af1d952e81c9eaa62783308f not found: ID does not exist"
Sep 17 22:11:18.464249 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:18.464216    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" path="/var/lib/kubelet/pods/6e727a2e-19a7-4ba6-ade7-785008b73897/volumes"
Sep 17 22:11:18.464726 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:18.464713    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" path="/var/lib/kubelet/pods/8ccb127e-b601-466e-bf4e-6f24ce34060d/volumes"
Sep 17 22:11:25.501203 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:25.501163    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 22:11:35.500474 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:35.500374    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 22:11:45.500583 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:45.500546    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 22:11:55.500983 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:11:55.500942    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 22:12:05.500501 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:05.500461    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 22:12:15.501526 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:15.501496    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:12:27.066309 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:27.066265    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"]
Sep 17 22:12:27.066762 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:27.066603    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kube-rbac-proxy" containerID="cri-o://77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65" gracePeriod=30
Sep 17 22:12:27.066762 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:27.066580    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container" containerID="cri-o://39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2" gracePeriod=30
Sep 17 22:12:27.869693 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:27.869657    2585 generic.go:356] "Generic (PLEG): container finished" podID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerID="77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65" exitCode=2
Sep 17 22:12:27.869894 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:27.869738    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" event={"ID":"b9243e38-c20b-4754-99a2-5b73d13420f5","Type":"ContainerDied","Data":"77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65"}
Sep 17 22:12:30.495983 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:30.495936    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 22:12:32.307471 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.307421    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:12:32.427998 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.427899    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9243e38-c20b-4754-99a2-5b73d13420f5-proxy-tls\") pod \"b9243e38-c20b-4754-99a2-5b73d13420f5\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") "
Sep 17 22:12:32.427998 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.427974    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-cabundle-cert\") pod \"b9243e38-c20b-4754-99a2-5b73d13420f5\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") "
Sep 17 22:12:32.428218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.428038    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-rbac-proxy-sar-config\") pod \"b9243e38-c20b-4754-99a2-5b73d13420f5\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") "
Sep 17 22:12:32.428218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.428059    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhf8w\" (UniqueName: \"kubernetes.io/projected/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-api-access-nhf8w\") pod \"b9243e38-c20b-4754-99a2-5b73d13420f5\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") "
Sep 17 22:12:32.428218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.428084    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9243e38-c20b-4754-99a2-5b73d13420f5-kserve-provision-location\") pod \"b9243e38-c20b-4754-99a2-5b73d13420f5\" (UID: \"b9243e38-c20b-4754-99a2-5b73d13420f5\") "
Sep 17 22:12:32.428379 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.428341    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b9243e38-c20b-4754-99a2-5b73d13420f5" (UID: "b9243e38-c20b-4754-99a2-5b73d13420f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:32.428496 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.428389    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9243e38-c20b-4754-99a2-5b73d13420f5" (UID: "b9243e38-c20b-4754-99a2-5b73d13420f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:32.428633 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.428534    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9243e38-c20b-4754-99a2-5b73d13420f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9243e38-c20b-4754-99a2-5b73d13420f5" (UID: "b9243e38-c20b-4754-99a2-5b73d13420f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:12:32.430316 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.430294    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9243e38-c20b-4754-99a2-5b73d13420f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9243e38-c20b-4754-99a2-5b73d13420f5" (UID: "b9243e38-c20b-4754-99a2-5b73d13420f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:12:32.430414 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.430392    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-api-access-nhf8w" (OuterVolumeSpecName: "kube-api-access-nhf8w") pod "b9243e38-c20b-4754-99a2-5b73d13420f5" (UID: "b9243e38-c20b-4754-99a2-5b73d13420f5"). InnerVolumeSpecName "kube-api-access-nhf8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:12:32.529214 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.529171    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9243e38-c20b-4754-99a2-5b73d13420f5-kserve-provision-location\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:12:32.529214 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.529206    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9243e38-c20b-4754-99a2-5b73d13420f5-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:12:32.529214 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.529216    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-cabundle-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:12:32.529214 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.529225    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:12:32.529510 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.529233    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhf8w\" (UniqueName: \"kubernetes.io/projected/b9243e38-c20b-4754-99a2-5b73d13420f5-kube-api-access-nhf8w\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:12:32.890015 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.889980    2585 generic.go:356] "Generic (PLEG): container finished" podID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerID="39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2" exitCode=0
Sep 17 22:12:32.890193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.890052    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"
Sep 17 22:12:32.890193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.890063    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" event={"ID":"b9243e38-c20b-4754-99a2-5b73d13420f5","Type":"ContainerDied","Data":"39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2"}
Sep 17 22:12:32.890193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.890131    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq" event={"ID":"b9243e38-c20b-4754-99a2-5b73d13420f5","Type":"ContainerDied","Data":"561d570a6726101de64d7c7effa612bdd3290a555ec6a2013312219470a68baf"}
Sep 17 22:12:32.890193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.890160    2585 scope.go:117] "RemoveContainer" containerID="77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65"
Sep 17 22:12:32.901123 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.901097    2585 scope.go:117] "RemoveContainer" containerID="39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2"
Sep 17 22:12:32.907573 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.907547    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"]
Sep 17 22:12:32.910240 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.910213    2585 scope.go:117] "RemoveContainer" containerID="89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b"
Sep 17 22:12:32.913574 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.913445    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-646f5bdbcb-dgzkq"]
Sep 17 22:12:32.924219 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.924197    2585 scope.go:117] "RemoveContainer" containerID="77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65"
Sep 17 22:12:32.924592 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:12:32.924571    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65\": container with ID starting with 77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65 not found: ID does not exist" containerID="77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65"
Sep 17 22:12:32.924650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.924602    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65"} err="failed to get container status \"77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65\": rpc error: code = NotFound desc = could not find container \"77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65\": container with ID starting with 77d9d62550af5838ac77be3901e32343e4d4c9e1d066f8f57cfa03f3fcd34c65 not found: ID does not exist"
Sep 17 22:12:32.924650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.924624    2585 scope.go:117] "RemoveContainer" containerID="39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2"
Sep 17 22:12:32.924914 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:12:32.924896    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2\": container with ID starting with 39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2 not found: ID does not exist" containerID="39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2"
Sep 17 22:12:32.924958 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.924919    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2"} err="failed to get container status \"39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2\": rpc error: code = NotFound desc = could not find container \"39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2\": container with ID starting with 39f917e2d2fbaf50d1643b2a2bf29a0ca9a625e624e0d89a75a1999a4db8dde2 not found: ID does not exist"
Sep 17 22:12:32.924958 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.924936    2585 scope.go:117] "RemoveContainer" containerID="89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b"
Sep 17 22:12:32.925169 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:12:32.925153    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b\": container with ID starting with 89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b not found: ID does not exist" containerID="89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b"
Sep 17 22:12:32.925210 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:32.925174    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b"} err="failed to get container status \"89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b\": rpc error: code = NotFound desc = could not find container \"89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b\": container with ID starting with 89eba382159df378eed26f010dbb7519d8ee3a3f4406391040e697f89cbeef5b not found: ID does not exist"
Sep 17 22:12:34.464146 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:34.464104    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" path="/var/lib/kubelet/pods/b9243e38-c20b-4754-99a2-5b73d13420f5/volumes"
Sep 17 22:12:37.307954 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.307918    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"]
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308242    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="storage-initializer"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308255    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="storage-initializer"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308265    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kube-rbac-proxy"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308271    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kube-rbac-proxy"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308277    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308283    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308289    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="storage-initializer"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308293    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="storage-initializer"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308302    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308307    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308318    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kube-rbac-proxy"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308323    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kube-rbac-proxy"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308329    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308334    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308340    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="storage-initializer"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308345    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="storage-initializer"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308358    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kube-rbac-proxy"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308363    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kube-rbac-proxy"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308407    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kube-rbac-proxy"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308414    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kube-rbac-proxy"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308421    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kserve-container"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308442    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9243e38-c20b-4754-99a2-5b73d13420f5" containerName="kserve-container"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308448    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e727a2e-19a7-4ba6-ade7-785008b73897" containerName="kube-rbac-proxy"
Sep 17 22:12:37.308489 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.308454    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ccb127e-b601-466e-bf4e-6f24ce34060d" containerName="kserve-container"
Sep 17 22:12:37.311495 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.311477    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.314039 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.314016    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:12:37.314185 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.314065    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:12:37.314185 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.314105    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ba0165-predictor-serving-cert\""
Sep 17 22:12:37.314185 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.314151    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:12:37.314922 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.314904    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:12:37.315020 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.314903    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:12:37.315020 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.314937    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ba0165-kube-rbac-proxy-sar-config\""
Sep 17 22:12:37.320842 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.320820    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"]
Sep 17 22:12:37.467614 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.467570    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t5bt\" (UniqueName: \"kubernetes.io/projected/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-api-access-4t5bt\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.467614 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.467612    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64cf994a-4f11-4205-b9b3-5757d6cae3ec-proxy-tls\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.467831 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.467639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.467831 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.467714    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-cabundle-cert\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.467831 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.467777    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kserve-provision-location\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.568254 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.568156    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-cabundle-cert\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.568464 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.568261    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kserve-provision-location\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.568464 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.568386    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t5bt\" (UniqueName: \"kubernetes.io/projected/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-api-access-4t5bt\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.568464 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.568442    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64cf994a-4f11-4205-b9b3-5757d6cae3ec-proxy-tls\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.568646 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.568535    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.568832 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.568800    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kserve-provision-location\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.569165 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.569145    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-cabundle-cert\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.569228 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.569145    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.571291 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.571270    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64cf994a-4f11-4205-b9b3-5757d6cae3ec-proxy-tls\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.577034 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.577005    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t5bt\" (UniqueName: \"kubernetes.io/projected/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-api-access-4t5bt\") pod \"isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") " pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.623026 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.622993    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:37.900062 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.899979    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"]
Sep 17 22:12:37.904147 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:12:37.904119    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cf994a_4f11_4205_b9b3_5757d6cae3ec.slice/crio-0f95acb509ad2e31fe9d620ab54c572b8b8ef9ddf922335029f00c69179705a6 WatchSource:0}: Error finding container 0f95acb509ad2e31fe9d620ab54c572b8b8ef9ddf922335029f00c69179705a6: Status 404 returned error can't find the container with id 0f95acb509ad2e31fe9d620ab54c572b8b8ef9ddf922335029f00c69179705a6
Sep 17 22:12:37.910899 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:37.910865    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" event={"ID":"64cf994a-4f11-4205-b9b3-5757d6cae3ec","Type":"ContainerStarted","Data":"0f95acb509ad2e31fe9d620ab54c572b8b8ef9ddf922335029f00c69179705a6"}
Sep 17 22:12:38.916840 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:38.916800    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" event={"ID":"64cf994a-4f11-4205-b9b3-5757d6cae3ec","Type":"ContainerStarted","Data":"1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6"}
Sep 17 22:12:42.933259 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:42.933218    2585 generic.go:356] "Generic (PLEG): container finished" podID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerID="1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6" exitCode=0
Sep 17 22:12:42.933801 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:42.933295    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" event={"ID":"64cf994a-4f11-4205-b9b3-5757d6cae3ec","Type":"ContainerDied","Data":"1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6"}
Sep 17 22:12:43.942624 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:43.942588    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" event={"ID":"64cf994a-4f11-4205-b9b3-5757d6cae3ec","Type":"ContainerStarted","Data":"6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514"}
Sep 17 22:12:43.943010 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:43.942663    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" event={"ID":"64cf994a-4f11-4205-b9b3-5757d6cae3ec","Type":"ContainerStarted","Data":"36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2"}
Sep 17 22:12:43.943010 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:43.942960    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:43.943077 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:43.943020    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:43.944497 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:43.944471    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:12:43.961808 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:43.961749    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podStartSLOduration=2.994009378 podStartE2EDuration="6.961729316s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.967719938s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:43.960234831 +0000 UTC m=+865.822488308" watchObservedRunningTime="2026-09-17 22:12:43.961729316 +0000 UTC m=+865.823982794"
Sep 17 22:12:44.946847 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:44.946799    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:12:49.950965 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:49.950929    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:12:49.951650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:49.951620    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:12:59.951545 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:12:59.951497    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:02.456262 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.456183    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"]
Sep 17 22:13:02.459817 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.459789    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.463120 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.463097    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-a3db6-predictor-serving-cert\""
Sep 17 22:13:02.463259 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.463147    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-a3db6-kube-rbac-proxy-sar-config\""
Sep 17 22:13:02.469456 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.469411    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"]
Sep 17 22:13:02.560595 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.560555    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9mj\" (UniqueName: \"kubernetes.io/projected/819327c3-3b02-409d-b8fc-a094c73824a9-kube-api-access-db9mj\") pod \"message-dumper-raw-a3db6-predictor-74d496889b-msg85\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") " pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.560803 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.560692    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/819327c3-3b02-409d-b8fc-a094c73824a9-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-a3db6-predictor-74d496889b-msg85\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") " pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.560803 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.560742    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/819327c3-3b02-409d-b8fc-a094c73824a9-proxy-tls\") pod \"message-dumper-raw-a3db6-predictor-74d496889b-msg85\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") " pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.661607 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.661564    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9mj\" (UniqueName: \"kubernetes.io/projected/819327c3-3b02-409d-b8fc-a094c73824a9-kube-api-access-db9mj\") pod \"message-dumper-raw-a3db6-predictor-74d496889b-msg85\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") " pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.661772 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.661631    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/819327c3-3b02-409d-b8fc-a094c73824a9-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-a3db6-predictor-74d496889b-msg85\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") " pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.661772 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.661665    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/819327c3-3b02-409d-b8fc-a094c73824a9-proxy-tls\") pod \"message-dumper-raw-a3db6-predictor-74d496889b-msg85\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") " pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.662454 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.662351    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/819327c3-3b02-409d-b8fc-a094c73824a9-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-a3db6-predictor-74d496889b-msg85\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") " pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.664312 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.664290    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/819327c3-3b02-409d-b8fc-a094c73824a9-proxy-tls\") pod \"message-dumper-raw-a3db6-predictor-74d496889b-msg85\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") " pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.670507 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.670482    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9mj\" (UniqueName: \"kubernetes.io/projected/819327c3-3b02-409d-b8fc-a094c73824a9-kube-api-access-db9mj\") pod \"message-dumper-raw-a3db6-predictor-74d496889b-msg85\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") " pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:02.771400 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:02.771360    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:03.047865 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:03.047831    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"]
Sep 17 22:13:03.050647 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:13:03.050612    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819327c3_3b02_409d_b8fc_a094c73824a9.slice/crio-b21a53fe987acec8b43e5ea53e95665aace87941515cde5638ee906edf698171 WatchSource:0}: Error finding container b21a53fe987acec8b43e5ea53e95665aace87941515cde5638ee906edf698171: Status 404 returned error can't find the container with id b21a53fe987acec8b43e5ea53e95665aace87941515cde5638ee906edf698171
Sep 17 22:13:04.023380 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:04.023342    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85" event={"ID":"819327c3-3b02-409d-b8fc-a094c73824a9","Type":"ContainerStarted","Data":"b21a53fe987acec8b43e5ea53e95665aace87941515cde5638ee906edf698171"}
Sep 17 22:13:05.029543 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:05.029503    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85" event={"ID":"819327c3-3b02-409d-b8fc-a094c73824a9","Type":"ContainerStarted","Data":"da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686"}
Sep 17 22:13:05.030012 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:05.029565    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85" event={"ID":"819327c3-3b02-409d-b8fc-a094c73824a9","Type":"ContainerStarted","Data":"39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1"}
Sep 17 22:13:05.030012 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:05.029641    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:05.046740 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:05.046686    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85" podStartSLOduration=1.900892104 podStartE2EDuration="3.046663857s" totalImagesPullingTime="1.145771753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:13:05.044778782 +0000 UTC m=+886.907032272" watchObservedRunningTime="2026-09-17 22:13:05.046663857 +0000 UTC m=+886.908917419"
Sep 17 22:13:06.033490 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:06.033421    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:06.035515 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:06.035492    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:09.951681 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:09.951640    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:13.047107 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:13.047079    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:13:19.952408 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:19.952365    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:20.407282 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:20.407250    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:13:20.408337 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:20.408316    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:13:29.952521 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:29.952476    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:39.951508 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:39.951468    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:49.951876 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:49.951834    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:13:57.459366 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:13:57.459317    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 22:14:07.460477 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:07.460419    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:14:50.576588 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:50.576506    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"]
Sep 17 22:14:50.577102 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:50.576781    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container" containerID="cri-o://36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2" gracePeriod=30
Sep 17 22:14:50.577102 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:50.576813    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kube-rbac-proxy" containerID="cri-o://6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514" gracePeriod=30
Sep 17 22:14:51.467827 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:51.467793    2585 generic.go:356] "Generic (PLEG): container finished" podID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerID="6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514" exitCode=2
Sep 17 22:14:51.467827 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:51.467831    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" event={"ID":"64cf994a-4f11-4205-b9b3-5757d6cae3ec","Type":"ContainerDied","Data":"6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514"}
Sep 17 22:14:54.947851 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:54.947800    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 22:14:55.937834 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.937766    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:14:55.987681 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.987651    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64cf994a-4f11-4205-b9b3-5757d6cae3ec-proxy-tls\") pod \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") "
Sep 17 22:14:55.987681 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.987686    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-rbac-proxy-sar-config\") pod \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") "
Sep 17 22:14:55.988197 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.987708    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t5bt\" (UniqueName: \"kubernetes.io/projected/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-api-access-4t5bt\") pod \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") "
Sep 17 22:14:55.988197 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.987740    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-cabundle-cert\") pod \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") "
Sep 17 22:14:55.988197 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.987762    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kserve-provision-location\") pod \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\" (UID: \"64cf994a-4f11-4205-b9b3-5757d6cae3ec\") "
Sep 17 22:14:55.988197 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.988144    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64cf994a-4f11-4205-b9b3-5757d6cae3ec" (UID: "64cf994a-4f11-4205-b9b3-5757d6cae3ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:14:55.988197 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.988153    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64cf994a-4f11-4205-b9b3-5757d6cae3ec" (UID: "64cf994a-4f11-4205-b9b3-5757d6cae3ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:14:55.988197 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.988148    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64cf994a-4f11-4205-b9b3-5757d6cae3ec" (UID: "64cf994a-4f11-4205-b9b3-5757d6cae3ec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:14:55.990022 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.989997    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-api-access-4t5bt" (OuterVolumeSpecName: "kube-api-access-4t5bt") pod "64cf994a-4f11-4205-b9b3-5757d6cae3ec" (UID: "64cf994a-4f11-4205-b9b3-5757d6cae3ec"). InnerVolumeSpecName "kube-api-access-4t5bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:14:55.990190 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:55.990173    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64cf994a-4f11-4205-b9b3-5757d6cae3ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64cf994a-4f11-4205-b9b3-5757d6cae3ec" (UID: "64cf994a-4f11-4205-b9b3-5757d6cae3ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:14:56.088291 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.088248    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64cf994a-4f11-4205-b9b3-5757d6cae3ec-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:14:56.088291 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.088284    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:14:56.088291 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.088299    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t5bt\" (UniqueName: \"kubernetes.io/projected/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kube-api-access-4t5bt\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:14:56.088584 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.088312    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cf994a-4f11-4205-b9b3-5757d6cae3ec-cabundle-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:14:56.088584 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.088323    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cf994a-4f11-4205-b9b3-5757d6cae3ec-kserve-provision-location\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:14:56.489953 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.489916    2585 generic.go:356] "Generic (PLEG): container finished" podID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerID="36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2" exitCode=0
Sep 17 22:14:56.490118 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.489971    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"
Sep 17 22:14:56.490118 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.489996    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" event={"ID":"64cf994a-4f11-4205-b9b3-5757d6cae3ec","Type":"ContainerDied","Data":"36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2"}
Sep 17 22:14:56.490118 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.490066    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj" event={"ID":"64cf994a-4f11-4205-b9b3-5757d6cae3ec","Type":"ContainerDied","Data":"0f95acb509ad2e31fe9d620ab54c572b8b8ef9ddf922335029f00c69179705a6"}
Sep 17 22:14:56.490221 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.490141    2585 scope.go:117] "RemoveContainer" containerID="6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514"
Sep 17 22:14:56.498735 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.498715    2585 scope.go:117] "RemoveContainer" containerID="36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2"
Sep 17 22:14:56.506691 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.506664    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"]
Sep 17 22:14:56.507151 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.507131    2585 scope.go:117] "RemoveContainer" containerID="1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6"
Sep 17 22:14:56.510277 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.510252    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba0165-predictor-5b8b769dfb-wfbmj"]
Sep 17 22:14:56.515751 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.515731    2585 scope.go:117] "RemoveContainer" containerID="6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514"
Sep 17 22:14:56.516088 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:14:56.516067    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514\": container with ID starting with 6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514 not found: ID does not exist" containerID="6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514"
Sep 17 22:14:56.516160 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.516097    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514"} err="failed to get container status \"6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514\": rpc error: code = NotFound desc = could not find container \"6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514\": container with ID starting with 6abc763999afe9d8a3ecd7879fd43f708c0c9713b223db6dfe028c299dd6f514 not found: ID does not exist"
Sep 17 22:14:56.516160 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.516116    2585 scope.go:117] "RemoveContainer" containerID="36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2"
Sep 17 22:14:56.516397 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:14:56.516382    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2\": container with ID starting with 36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2 not found: ID does not exist" containerID="36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2"
Sep 17 22:14:56.516465 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.516400    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2"} err="failed to get container status \"36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2\": rpc error: code = NotFound desc = could not find container \"36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2\": container with ID starting with 36a06d7212c1b5aa029b81678e7527efedce6dcc586da7c051ac87d761737ad2 not found: ID does not exist"
Sep 17 22:14:56.516465 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.516412    2585 scope.go:117] "RemoveContainer" containerID="1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6"
Sep 17 22:14:56.516646 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:14:56.516628    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6\": container with ID starting with 1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6 not found: ID does not exist" containerID="1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6"
Sep 17 22:14:56.516689 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:56.516650    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6"} err="failed to get container status \"1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6\": rpc error: code = NotFound desc = could not find container \"1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6\": container with ID starting with 1971bf2982dcf04b332320126891fdeadaf431f3d2eadff8d04900c00c5e3ef6 not found: ID does not exist"
Sep 17 22:14:57.484720 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.484690    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-a3db6-predictor-74d496889b-msg85_819327c3-3b02-409d-b8fc-a094c73824a9/kserve-container/0.log"
Sep 17 22:14:57.675078 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.675036    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"]
Sep 17 22:14:57.675350 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.675325    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85" podUID="819327c3-3b02-409d-b8fc-a094c73824a9" containerName="kserve-container" containerID="cri-o://39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1" gracePeriod=30
Sep 17 22:14:57.675453 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.675399    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85" podUID="819327c3-3b02-409d-b8fc-a094c73824a9" containerName="kube-rbac-proxy" containerID="cri-o://da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686" gracePeriod=30
Sep 17 22:14:57.774155 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.774116    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"]
Sep 17 22:14:57.774415 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.774403    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="storage-initializer"
Sep 17 22:14:57.774479 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.774417    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="storage-initializer"
Sep 17 22:14:57.774479 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.774450    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container"
Sep 17 22:14:57.774479 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.774456    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container"
Sep 17 22:14:57.774479 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.774469    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kube-rbac-proxy"
Sep 17 22:14:57.774479 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.774474    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kube-rbac-proxy"
Sep 17 22:14:57.774631 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.774520    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kube-rbac-proxy"
Sep 17 22:14:57.774631 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.774528    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" containerName="kserve-container"
Sep 17 22:14:57.777638 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.777619    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.780365 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.780266    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:14:57.780365 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.780300    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-4f1a2-predictor-serving-cert\""
Sep 17 22:14:57.780706 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.780681    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-4f1a2-kube-rbac-proxy-sar-config\""
Sep 17 22:14:57.780901 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.780707    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:14:57.794886 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.794859    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"]
Sep 17 22:14:57.799925 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.799895    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5561da14-2f61-4586-817a-febf36d9bd7b-proxy-tls\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.800065 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.799930    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsfk9\" (UniqueName: \"kubernetes.io/projected/5561da14-2f61-4586-817a-febf36d9bd7b-kube-api-access-zsfk9\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.800065 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.799962    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5561da14-2f61-4586-817a-febf36d9bd7b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.800065 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.799999    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-cabundle-cert\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.800065 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.800017    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.900543 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.900493    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-cabundle-cert\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.900543 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.900544    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.900770 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.900633    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5561da14-2f61-4586-817a-febf36d9bd7b-proxy-tls\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.900770 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.900664    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsfk9\" (UniqueName: \"kubernetes.io/projected/5561da14-2f61-4586-817a-febf36d9bd7b-kube-api-access-zsfk9\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.900770 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.900694    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5561da14-2f61-4586-817a-febf36d9bd7b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.900936 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:14:57.900808    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-serving-cert: secret "isvc-sklearn-scale-raw-4f1a2-predictor-serving-cert" not found
Sep 17 22:14:57.900936 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:14:57.900885    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5561da14-2f61-4586-817a-febf36d9bd7b-proxy-tls podName:5561da14-2f61-4586-817a-febf36d9bd7b nodeName:}" failed. No retries permitted until 2026-09-17 22:14:58.400864123 +0000 UTC m=+1000.263117578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5561da14-2f61-4586-817a-febf36d9bd7b-proxy-tls") pod "isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" (UID: "5561da14-2f61-4586-817a-febf36d9bd7b") : secret "isvc-sklearn-scale-raw-4f1a2-predictor-serving-cert" not found
Sep 17 22:14:57.901103 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.901065    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5561da14-2f61-4586-817a-febf36d9bd7b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.901260 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.901242    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-cabundle-cert\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.901313 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.901262    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.909525 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.909495    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsfk9\" (UniqueName: \"kubernetes.io/projected/5561da14-2f61-4586-817a-febf36d9bd7b-kube-api-access-zsfk9\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:57.935090 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:57.935056    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:14:58.001021 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.000981    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/819327c3-3b02-409d-b8fc-a094c73824a9-proxy-tls\") pod \"819327c3-3b02-409d-b8fc-a094c73824a9\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") "
Sep 17 22:14:58.001232 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.001049    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db9mj\" (UniqueName: \"kubernetes.io/projected/819327c3-3b02-409d-b8fc-a094c73824a9-kube-api-access-db9mj\") pod \"819327c3-3b02-409d-b8fc-a094c73824a9\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") "
Sep 17 22:14:58.001232 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.001074    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/819327c3-3b02-409d-b8fc-a094c73824a9-kube-rbac-proxy-sar-config\") pod \"819327c3-3b02-409d-b8fc-a094c73824a9\" (UID: \"819327c3-3b02-409d-b8fc-a094c73824a9\") "
Sep 17 22:14:58.001535 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.001506    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819327c3-3b02-409d-b8fc-a094c73824a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "819327c3-3b02-409d-b8fc-a094c73824a9" (UID: "819327c3-3b02-409d-b8fc-a094c73824a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:14:58.003392 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.003367    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819327c3-3b02-409d-b8fc-a094c73824a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "819327c3-3b02-409d-b8fc-a094c73824a9" (UID: "819327c3-3b02-409d-b8fc-a094c73824a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:14:58.003503 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.003483    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819327c3-3b02-409d-b8fc-a094c73824a9-kube-api-access-db9mj" (OuterVolumeSpecName: "kube-api-access-db9mj") pod "819327c3-3b02-409d-b8fc-a094c73824a9" (UID: "819327c3-3b02-409d-b8fc-a094c73824a9"). InnerVolumeSpecName "kube-api-access-db9mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:14:58.101932 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.101842    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db9mj\" (UniqueName: \"kubernetes.io/projected/819327c3-3b02-409d-b8fc-a094c73824a9-kube-api-access-db9mj\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:14:58.101932 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.101872    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/819327c3-3b02-409d-b8fc-a094c73824a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:14:58.101932 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.101883    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/819327c3-3b02-409d-b8fc-a094c73824a9-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:14:58.404258 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.404170    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5561da14-2f61-4586-817a-febf36d9bd7b-proxy-tls\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:58.406807 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.406766    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5561da14-2f61-4586-817a-febf36d9bd7b-proxy-tls\") pod \"isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:58.464793 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.464764    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64cf994a-4f11-4205-b9b3-5757d6cae3ec" path="/var/lib/kubelet/pods/64cf994a-4f11-4205-b9b3-5757d6cae3ec/volumes"
Sep 17 22:14:58.499700 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.499658    2585 generic.go:356] "Generic (PLEG): container finished" podID="819327c3-3b02-409d-b8fc-a094c73824a9" containerID="da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686" exitCode=2
Sep 17 22:14:58.499700 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.499685    2585 generic.go:356] "Generic (PLEG): container finished" podID="819327c3-3b02-409d-b8fc-a094c73824a9" containerID="39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1" exitCode=2
Sep 17 22:14:58.500142 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.499708    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85" event={"ID":"819327c3-3b02-409d-b8fc-a094c73824a9","Type":"ContainerDied","Data":"da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686"}
Sep 17 22:14:58.500142 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.499758    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85" event={"ID":"819327c3-3b02-409d-b8fc-a094c73824a9","Type":"ContainerDied","Data":"39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1"}
Sep 17 22:14:58.500142 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.499780    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85" event={"ID":"819327c3-3b02-409d-b8fc-a094c73824a9","Type":"ContainerDied","Data":"b21a53fe987acec8b43e5ea53e95665aace87941515cde5638ee906edf698171"}
Sep 17 22:14:58.500142 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.499712    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"
Sep 17 22:14:58.500142 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.499814    2585 scope.go:117] "RemoveContainer" containerID="da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686"
Sep 17 22:14:58.508352 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.508330    2585 scope.go:117] "RemoveContainer" containerID="39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1"
Sep 17 22:14:58.517130 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.517105    2585 scope.go:117] "RemoveContainer" containerID="da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686"
Sep 17 22:14:58.517487 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:14:58.517467    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686\": container with ID starting with da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686 not found: ID does not exist" containerID="da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686"
Sep 17 22:14:58.517556 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.517498    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686"} err="failed to get container status \"da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686\": rpc error: code = NotFound desc = could not find container \"da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686\": container with ID starting with da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686 not found: ID does not exist"
Sep 17 22:14:58.517556 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.517517    2585 scope.go:117] "RemoveContainer" containerID="39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1"
Sep 17 22:14:58.517801 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:14:58.517783    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1\": container with ID starting with 39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1 not found: ID does not exist" containerID="39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1"
Sep 17 22:14:58.517850 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.517805    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1"} err="failed to get container status \"39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1\": rpc error: code = NotFound desc = could not find container \"39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1\": container with ID starting with 39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1 not found: ID does not exist"
Sep 17 22:14:58.517850 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.517811    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"]
Sep 17 22:14:58.517850 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.517819    2585 scope.go:117] "RemoveContainer" containerID="da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686"
Sep 17 22:14:58.518110 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.518092    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686"} err="failed to get container status \"da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686\": rpc error: code = NotFound desc = could not find container \"da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686\": container with ID starting with da35ad72c49b05e0b8d7e5c6fa52f75f554266e3b49cbe5d7519301849303686 not found: ID does not exist"
Sep 17 22:14:58.518148 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.518110    2585 scope.go:117] "RemoveContainer" containerID="39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1"
Sep 17 22:14:58.518356 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.518338    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1"} err="failed to get container status \"39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1\": rpc error: code = NotFound desc = could not find container \"39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1\": container with ID starting with 39edcda4d87c190773955f4ecc4af109e1a2a37d417ef7a28019767fe73200d1 not found: ID does not exist"
Sep 17 22:14:58.521122 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.521097    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-a3db6-predictor-74d496889b-msg85"]
Sep 17 22:14:58.688878 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.688770    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:14:58.968027 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:58.968002    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"]
Sep 17 22:14:58.970499 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:14:58.970472    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5561da14_2f61_4586_817a_febf36d9bd7b.slice/crio-a29d061fce82f29a73bf095eff170f684ac74fc29c708ceb73428f100ec2231b WatchSource:0}: Error finding container a29d061fce82f29a73bf095eff170f684ac74fc29c708ceb73428f100ec2231b: Status 404 returned error can't find the container with id a29d061fce82f29a73bf095eff170f684ac74fc29c708ceb73428f100ec2231b
Sep 17 22:14:59.505591 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:59.505552    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" event={"ID":"5561da14-2f61-4586-817a-febf36d9bd7b","Type":"ContainerStarted","Data":"0c34591a5757251b68e66590b52da9c6d9f93c95f5404a6a91143e115bf76c24"}
Sep 17 22:14:59.506010 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:14:59.505611    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" event={"ID":"5561da14-2f61-4586-817a-febf36d9bd7b","Type":"ContainerStarted","Data":"a29d061fce82f29a73bf095eff170f684ac74fc29c708ceb73428f100ec2231b"}
Sep 17 22:15:00.463736 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:00.463702    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="819327c3-3b02-409d-b8fc-a094c73824a9" path="/var/lib/kubelet/pods/819327c3-3b02-409d-b8fc-a094c73824a9/volumes"
Sep 17 22:15:02.793818 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:15:02.793777    2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5561da14_2f61_4586_817a_febf36d9bd7b.slice/crio-0c34591a5757251b68e66590b52da9c6d9f93c95f5404a6a91143e115bf76c24.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5561da14_2f61_4586_817a_febf36d9bd7b.slice/crio-conmon-0c34591a5757251b68e66590b52da9c6d9f93c95f5404a6a91143e115bf76c24.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 22:15:03.522841 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:03.522802    2585 generic.go:356] "Generic (PLEG): container finished" podID="5561da14-2f61-4586-817a-febf36d9bd7b" containerID="0c34591a5757251b68e66590b52da9c6d9f93c95f5404a6a91143e115bf76c24" exitCode=0
Sep 17 22:15:03.523051 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:03.522873    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" event={"ID":"5561da14-2f61-4586-817a-febf36d9bd7b","Type":"ContainerDied","Data":"0c34591a5757251b68e66590b52da9c6d9f93c95f5404a6a91143e115bf76c24"}
Sep 17 22:15:04.529405 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:04.529363    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" event={"ID":"5561da14-2f61-4586-817a-febf36d9bd7b","Type":"ContainerStarted","Data":"6997b15d07ee6fce80d6f73c91074e5b8a4b28145545ab4a126d58da6be87f78"}
Sep 17 22:15:04.529805 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:04.529445    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" event={"ID":"5561da14-2f61-4586-817a-febf36d9bd7b","Type":"ContainerStarted","Data":"024caf414fa94e2e03d1803891190d8e7f0cc71b8ad6672c96eb4f067d479fdb"}
Sep 17 22:15:04.529805 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:04.529702    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:15:04.529805 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:04.529756    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:15:04.531280 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:04.531250    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:15:04.547836 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:04.547786    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podStartSLOduration=3.820090203 podStartE2EDuration="7.547770902s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.727680699s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:15:04.546121355 +0000 UTC m=+1006.408374849" watchObservedRunningTime="2026-09-17 22:15:04.547770902 +0000 UTC m=+1006.410024377"
Sep 17 22:15:05.534060 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:05.534023    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:15:10.538625 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.538596    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:15:10.539227 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.539199    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:15:10.987657 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.987566    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"]
Sep 17 22:15:10.987890 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.987877    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="819327c3-3b02-409d-b8fc-a094c73824a9" containerName="kserve-container"
Sep 17 22:15:10.987932 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.987891    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="819327c3-3b02-409d-b8fc-a094c73824a9" containerName="kserve-container"
Sep 17 22:15:10.987932 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.987903    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="819327c3-3b02-409d-b8fc-a094c73824a9" containerName="kube-rbac-proxy"
Sep 17 22:15:10.987932 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.987908    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="819327c3-3b02-409d-b8fc-a094c73824a9" containerName="kube-rbac-proxy"
Sep 17 22:15:10.988030 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.987976    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="819327c3-3b02-409d-b8fc-a094c73824a9" containerName="kserve-container"
Sep 17 22:15:10.988030 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.987986    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="819327c3-3b02-409d-b8fc-a094c73824a9" containerName="kube-rbac-proxy"
Sep 17 22:15:10.990939 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.990916    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:10.993411 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.993388    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fab04-predictor-serving-cert\""
Sep 17 22:15:10.993722 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:10.993697    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fab04-kube-rbac-proxy-sar-config\""
Sep 17 22:15:11.002865 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.002832    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"]
Sep 17 22:15:11.097157 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.097106    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8nd\" (UniqueName: \"kubernetes.io/projected/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-api-access-9z8nd\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.097157 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.097154    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kserve-provision-location\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.097382 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.097182    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-cabundle-cert\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.097382 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.097260    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.097382 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.097341    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-proxy-tls\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.198319 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.198283    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-proxy-tls\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.198481 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.198366    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8nd\" (UniqueName: \"kubernetes.io/projected/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-api-access-9z8nd\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.198481 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.198393    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kserve-provision-location\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.198481 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.198421    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-cabundle-cert\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.198677 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.198485    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.198914 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.198892    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kserve-provision-location\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.199262 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.199243    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.199304 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.199245    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-cabundle-cert\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.201080 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.201058    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-proxy-tls\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.206954 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.206931    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8nd\" (UniqueName: \"kubernetes.io/projected/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-api-access-9z8nd\") pod \"raw-sklearn-fab04-predictor-5dc968464f-2jg29\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") " pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.300773 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.300733    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:11.573795 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:11.573762    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"]
Sep 17 22:15:11.576440 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:15:11.576389    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc4bd9d_c4f2_40f4_96e5_46e2d3053350.slice/crio-a1776c9008ce0940f7ed892dbc9131053c78541bc97b364928c1a68bd92748f2 WatchSource:0}: Error finding container a1776c9008ce0940f7ed892dbc9131053c78541bc97b364928c1a68bd92748f2: Status 404 returned error can't find the container with id a1776c9008ce0940f7ed892dbc9131053c78541bc97b364928c1a68bd92748f2
Sep 17 22:15:12.563139 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:12.563103    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" event={"ID":"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350","Type":"ContainerStarted","Data":"f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157"}
Sep 17 22:15:12.563305 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:12.563160    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" event={"ID":"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350","Type":"ContainerStarted","Data":"a1776c9008ce0940f7ed892dbc9131053c78541bc97b364928c1a68bd92748f2"}
Sep 17 22:15:15.576732 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:15.576694    2585 generic.go:356] "Generic (PLEG): container finished" podID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerID="f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157" exitCode=0
Sep 17 22:15:15.577121 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:15.576775    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" event={"ID":"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350","Type":"ContainerDied","Data":"f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157"}
Sep 17 22:15:16.583794 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:16.583755    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" event={"ID":"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350","Type":"ContainerStarted","Data":"cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc"}
Sep 17 22:15:16.584309 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:16.583836    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" event={"ID":"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350","Type":"ContainerStarted","Data":"c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644"}
Sep 17 22:15:16.584309 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:16.584064    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:16.607137 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:16.607086    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podStartSLOduration=2.978636325 podStartE2EDuration="6.607070276s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.628433951s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:15:16.605318468 +0000 UTC m=+1018.467571941" watchObservedRunningTime="2026-09-17 22:15:16.607070276 +0000 UTC m=+1018.469323751"
Sep 17 22:15:17.588211 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:17.588171    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:17.589816 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:17.589788    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:15:18.592993 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:18.592952    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:15:20.539490 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:20.539455    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:15:23.597826 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:23.597794    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:15:23.598329 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:23.598290    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:15:30.539915 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:30.539866    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:15:33.598973 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:33.598927    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:15:40.540037 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:40.539996    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:15:43.599303 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:43.599256    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:15:50.539979 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:50.539933    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:15:53.598698 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:15:53.598645    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:16:00.539082 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:00.539047    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:16:03.598479 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:03.598413    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:16:10.539925 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:10.539881    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:16:13.598289 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:13.598248    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:16:20.539939 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:20.539900    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:16:23.598706 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:23.598666    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:16:30.539065 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:30.539020    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:16:33.599606 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:33.599568    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:16:36.459554 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:36.459517    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:16:41.107524 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:41.107487    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"]
Sep 17 22:16:41.108035 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:41.107753    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" containerID="cri-o://c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644" gracePeriod=30
Sep 17 22:16:41.108035 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:41.107803    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kube-rbac-proxy" containerID="cri-o://cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc" gracePeriod=30
Sep 17 22:16:41.933218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:41.933167    2585 generic.go:356] "Generic (PLEG): container finished" podID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerID="cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc" exitCode=2
Sep 17 22:16:41.933394 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:41.933245    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" event={"ID":"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350","Type":"ContainerDied","Data":"cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc"}
Sep 17 22:16:43.593382 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:43.593334    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 22:16:43.598397 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:43.598357    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 22:16:46.459582 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.459546    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:16:46.461987 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.461956    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:16:46.576650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.576547    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-rbac-proxy-sar-config\") pod \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") "
Sep 17 22:16:46.576650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.576633    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kserve-provision-location\") pod \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") "
Sep 17 22:16:46.576881 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.576699    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-proxy-tls\") pod \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") "
Sep 17 22:16:46.576881 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.576764    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z8nd\" (UniqueName: \"kubernetes.io/projected/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-api-access-9z8nd\") pod \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") "
Sep 17 22:16:46.576881 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.576792    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-cabundle-cert\") pod \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\" (UID: \"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350\") "
Sep 17 22:16:46.577019 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.576959    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" (UID: "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:16:46.577133 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.577000    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" (UID: "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:16:46.577172 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.577153    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" (UID: "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:16:46.579080 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.579056    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" (UID: "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:16:46.579141 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.579099    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-api-access-9z8nd" (OuterVolumeSpecName: "kube-api-access-9z8nd") pod "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" (UID: "1dc4bd9d-c4f2-40f4-96e5-46e2d3053350"). InnerVolumeSpecName "kube-api-access-9z8nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:16:46.677628 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.677587    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z8nd\" (UniqueName: \"kubernetes.io/projected/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-api-access-9z8nd\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:16:46.677628 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.677621    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-cabundle-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:16:46.677628 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.677630    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:16:46.677628 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.677638    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-kserve-provision-location\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:16:46.677884 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.677648    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:16:46.954805 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.954713    2585 generic.go:356] "Generic (PLEG): container finished" podID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerID="c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644" exitCode=0
Sep 17 22:16:46.954957 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.954794    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" event={"ID":"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350","Type":"ContainerDied","Data":"c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644"}
Sep 17 22:16:46.954957 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.954811    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"
Sep 17 22:16:46.954957 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.954895    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29" event={"ID":"1dc4bd9d-c4f2-40f4-96e5-46e2d3053350","Type":"ContainerDied","Data":"a1776c9008ce0940f7ed892dbc9131053c78541bc97b364928c1a68bd92748f2"}
Sep 17 22:16:46.954957 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.954926    2585 scope.go:117] "RemoveContainer" containerID="cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc"
Sep 17 22:16:46.963809 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.963790    2585 scope.go:117] "RemoveContainer" containerID="c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644"
Sep 17 22:16:46.971927 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.971905    2585 scope.go:117] "RemoveContainer" containerID="f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157"
Sep 17 22:16:46.977786 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.977762    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"]
Sep 17 22:16:46.980810 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.980791    2585 scope.go:117] "RemoveContainer" containerID="cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc"
Sep 17 22:16:46.981178 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:16:46.981160    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc\": container with ID starting with cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc not found: ID does not exist" containerID="cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc"
Sep 17 22:16:46.981229 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.981189    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc"} err="failed to get container status \"cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc\": rpc error: code = NotFound desc = could not find container \"cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc\": container with ID starting with cdfc0ecf1219aef84b7314b6f1da588acde554502c3867b70b9f0aa751b389cc not found: ID does not exist"
Sep 17 22:16:46.981229 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.981208    2585 scope.go:117] "RemoveContainer" containerID="c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644"
Sep 17 22:16:46.981518 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:16:46.981499    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644\": container with ID starting with c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644 not found: ID does not exist" containerID="c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644"
Sep 17 22:16:46.981579 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.981527    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644"} err="failed to get container status \"c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644\": rpc error: code = NotFound desc = could not find container \"c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644\": container with ID starting with c8d956b43eca866917af5b0a78a5f29bc424c263796a9f369b420461e4eb0644 not found: ID does not exist"
Sep 17 22:16:46.981579 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.981546    2585 scope.go:117] "RemoveContainer" containerID="f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157"
Sep 17 22:16:46.981779 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:16:46.981762    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157\": container with ID starting with f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157 not found: ID does not exist" containerID="f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157"
Sep 17 22:16:46.981821 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.981783    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157"} err="failed to get container status \"f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157\": rpc error: code = NotFound desc = could not find container \"f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157\": container with ID starting with f3763ac8000d165ca4127136b3327c16421d4383752b16881aee572d86132157 not found: ID does not exist"
Sep 17 22:16:46.981866 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:46.981848    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fab04-predictor-5dc968464f-2jg29"]
Sep 17 22:16:48.463616 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:48.463583    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" path="/var/lib/kubelet/pods/1dc4bd9d-c4f2-40f4-96e5-46e2d3053350/volumes"
Sep 17 22:16:56.459794 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:16:56.459750    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:06.460204 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:06.460154    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:16.460218 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:16.460173    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:26.459998 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:26.459943    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:36.460142 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:36.460036    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:46.463682 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:46.463631    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:17:47.876660 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:47.876622    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"]
Sep 17 22:17:47.877134 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:47.877058    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" containerID="cri-o://024caf414fa94e2e03d1803891190d8e7f0cc71b8ad6672c96eb4f067d479fdb" gracePeriod=30
Sep 17 22:17:47.877418 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:47.877365    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kube-rbac-proxy" containerID="cri-o://6997b15d07ee6fce80d6f73c91074e5b8a4b28145545ab4a126d58da6be87f78" gracePeriod=30
Sep 17 22:17:48.203792 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:48.203704    2585 generic.go:356] "Generic (PLEG): container finished" podID="5561da14-2f61-4586-817a-febf36d9bd7b" containerID="6997b15d07ee6fce80d6f73c91074e5b8a4b28145545ab4a126d58da6be87f78" exitCode=2
Sep 17 22:17:48.203947 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:48.203781    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" event={"ID":"5561da14-2f61-4586-817a-febf36d9bd7b","Type":"ContainerDied","Data":"6997b15d07ee6fce80d6f73c91074e5b8a4b28145545ab4a126d58da6be87f78"}
Sep 17 22:17:50.534820 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:50.534760    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 22:17:53.142934 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.142888    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"]
Sep 17 22:17:53.143420 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.143304    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kube-rbac-proxy"
Sep 17 22:17:53.143420 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.143322    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kube-rbac-proxy"
Sep 17 22:17:53.143420 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.143342    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="storage-initializer"
Sep 17 22:17:53.143420 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.143350    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="storage-initializer"
Sep 17 22:17:53.143420 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.143394    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container"
Sep 17 22:17:53.143420 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.143403    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container"
Sep 17 22:17:53.143772 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.143505    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kserve-container"
Sep 17 22:17:53.143772 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.143520    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc4bd9d-c4f2-40f4-96e5-46e2d3053350" containerName="kube-rbac-proxy"
Sep 17 22:17:53.146723 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.146700    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.148923 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.148885    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 22:17:53.149060 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.148984    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 22:17:53.155672 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.155636    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"]
Sep 17 22:17:53.206039 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.205997    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.206233 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.206066    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.206233 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.206123    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrgp5\" (UniqueName: \"kubernetes.io/projected/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-api-access-nrgp5\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.206233 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.206152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.206233 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.206172    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.306532 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.306490    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.306532 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.306532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrgp5\" (UniqueName: \"kubernetes.io/projected/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-api-access-nrgp5\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.306806 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.306553    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.306806 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.306573    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.306806 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.306597    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.307113 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.307072    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.307302 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.307284    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.307373 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.307349    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.309392 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.309366    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.314043 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.314014    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrgp5\" (UniqueName: \"kubernetes.io/projected/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-api-access-nrgp5\") pod \"isvc-canary-create-predictor-5f845bd94d-j7gbv\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.458633 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.458531    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:53.734719 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:53.734629    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"]
Sep 17 22:17:53.738064 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:17:53.738035    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdaa498_b6fb_4912_a4f7_24ffa6e382c1.slice/crio-b92011f907a2e040ad8bcc2090f1f53d2d5d95e875367af8bf16d46a8bb95f0f WatchSource:0}: Error finding container b92011f907a2e040ad8bcc2090f1f53d2d5d95e875367af8bf16d46a8bb95f0f: Status 404 returned error can't find the container with id b92011f907a2e040ad8bcc2090f1f53d2d5d95e875367af8bf16d46a8bb95f0f
Sep 17 22:17:54.228002 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:54.227956    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" event={"ID":"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1","Type":"ContainerStarted","Data":"661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29"}
Sep 17 22:17:54.228396 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:54.228077    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" event={"ID":"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1","Type":"ContainerStarted","Data":"b92011f907a2e040ad8bcc2090f1f53d2d5d95e875367af8bf16d46a8bb95f0f"}
Sep 17 22:17:55.534791 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:55.534737    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 22:17:56.459468 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:56.459406    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 22:17:58.248182 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.248107    2585 generic.go:356] "Generic (PLEG): container finished" podID="5561da14-2f61-4586-817a-febf36d9bd7b" containerID="024caf414fa94e2e03d1803891190d8e7f0cc71b8ad6672c96eb4f067d479fdb" exitCode=0
Sep 17 22:17:58.248619 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.248266    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" event={"ID":"5561da14-2f61-4586-817a-febf36d9bd7b","Type":"ContainerDied","Data":"024caf414fa94e2e03d1803891190d8e7f0cc71b8ad6672c96eb4f067d479fdb"}
Sep 17 22:17:58.250193 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.250162    2585 generic.go:356] "Generic (PLEG): container finished" podID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerID="661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29" exitCode=0
Sep 17 22:17:58.250338 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.250209    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" event={"ID":"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1","Type":"ContainerDied","Data":"661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29"}
Sep 17 22:17:58.320092 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.320061    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:17:58.451328 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.451286    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5561da14-2f61-4586-817a-febf36d9bd7b-proxy-tls\") pod \"5561da14-2f61-4586-817a-febf36d9bd7b\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") "
Sep 17 22:17:58.451552 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.451359    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsfk9\" (UniqueName: \"kubernetes.io/projected/5561da14-2f61-4586-817a-febf36d9bd7b-kube-api-access-zsfk9\") pod \"5561da14-2f61-4586-817a-febf36d9bd7b\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") "
Sep 17 22:17:58.451552 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.451396    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-cabundle-cert\") pod \"5561da14-2f61-4586-817a-febf36d9bd7b\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") "
Sep 17 22:17:58.451552 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.451456    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5561da14-2f61-4586-817a-febf36d9bd7b-kserve-provision-location\") pod \"5561da14-2f61-4586-817a-febf36d9bd7b\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") "
Sep 17 22:17:58.451552 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.451485    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-kube-rbac-proxy-sar-config\") pod \"5561da14-2f61-4586-817a-febf36d9bd7b\" (UID: \"5561da14-2f61-4586-817a-febf36d9bd7b\") "
Sep 17 22:17:58.451845 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.451803    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5561da14-2f61-4586-817a-febf36d9bd7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5561da14-2f61-4586-817a-febf36d9bd7b" (UID: "5561da14-2f61-4586-817a-febf36d9bd7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:58.451969 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.451844    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5561da14-2f61-4586-817a-febf36d9bd7b" (UID: "5561da14-2f61-4586-817a-febf36d9bd7b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:17:58.451969 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.451849    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5561da14-2f61-4586-817a-febf36d9bd7b" (UID: "5561da14-2f61-4586-817a-febf36d9bd7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:17:58.453854 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.453832    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5561da14-2f61-4586-817a-febf36d9bd7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5561da14-2f61-4586-817a-febf36d9bd7b" (UID: "5561da14-2f61-4586-817a-febf36d9bd7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:17:58.453936 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.453862    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5561da14-2f61-4586-817a-febf36d9bd7b-kube-api-access-zsfk9" (OuterVolumeSpecName: "kube-api-access-zsfk9") pod "5561da14-2f61-4586-817a-febf36d9bd7b" (UID: "5561da14-2f61-4586-817a-febf36d9bd7b"). InnerVolumeSpecName "kube-api-access-zsfk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:17:58.552994 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.552955    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-cabundle-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:17:58.552994 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.552987    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5561da14-2f61-4586-817a-febf36d9bd7b-kserve-provision-location\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:17:58.552994 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.552996    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5561da14-2f61-4586-817a-febf36d9bd7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:17:58.552994 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.553007    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5561da14-2f61-4586-817a-febf36d9bd7b-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:17:58.553263 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:58.553016    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsfk9\" (UniqueName: \"kubernetes.io/projected/5561da14-2f61-4586-817a-febf36d9bd7b-kube-api-access-zsfk9\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:17:59.257024 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.256986    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" event={"ID":"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1","Type":"ContainerStarted","Data":"e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a"}
Sep 17 22:17:59.257464 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.257054    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" event={"ID":"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1","Type":"ContainerStarted","Data":"206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f"}
Sep 17 22:17:59.257464 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.257421    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:59.257632 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.257495    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:17:59.259199 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.259168    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv" event={"ID":"5561da14-2f61-4586-817a-febf36d9bd7b","Type":"ContainerDied","Data":"a29d061fce82f29a73bf095eff170f684ac74fc29c708ceb73428f100ec2231b"}
Sep 17 22:17:59.259331 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.259198    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"
Sep 17 22:17:59.259331 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.259233    2585 scope.go:117] "RemoveContainer" containerID="6997b15d07ee6fce80d6f73c91074e5b8a4b28145545ab4a126d58da6be87f78"
Sep 17 22:17:59.259407 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.259322    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 22:17:59.269557 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.269526    2585 scope.go:117] "RemoveContainer" containerID="024caf414fa94e2e03d1803891190d8e7f0cc71b8ad6672c96eb4f067d479fdb"
Sep 17 22:17:59.274871 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.274829    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podStartSLOduration=2.622088217 podStartE2EDuration="6.274811956s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.652723739s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:17:59.274684941 +0000 UTC m=+1181.136938452" watchObservedRunningTime="2026-09-17 22:17:59.274811956 +0000 UTC m=+1181.137065482"
Sep 17 22:17:59.282019 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.280459    2585 scope.go:117] "RemoveContainer" containerID="0c34591a5757251b68e66590b52da9c6d9f93c95f5404a6a91143e115bf76c24"
Sep 17 22:17:59.288078 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.288047    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"]
Sep 17 22:17:59.293721 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:17:59.293695    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-4f1a2-predictor-5ff4b8f5ff-r2fwv"]
Sep 17 22:18:00.264859 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:00.264827    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 22:18:00.464658 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:00.464621    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" path="/var/lib/kubelet/pods/5561da14-2f61-4586-817a-febf36d9bd7b/volumes"
Sep 17 22:18:05.269764 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:05.269737    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:18:05.270402 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:05.270373    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 22:18:15.271317 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:15.271275    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 22:18:20.432135 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:20.432104    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:18:20.433785 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:20.433760    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:18:25.271365 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:25.271322    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 22:18:35.270907 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:35.270857    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 22:18:45.271031 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:45.270987    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 22:18:55.271120 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:18:55.271076    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 22:19:05.271014 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:05.270931    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 22:19:14.739752 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:14.739716    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"]
Sep 17 22:19:14.740163 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:14.739968    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container" containerID="cri-o://206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f" gracePeriod=30
Sep 17 22:19:14.740163 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:14.740103    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kube-rbac-proxy" containerID="cri-o://e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a" gracePeriod=30
Sep 17 22:19:14.740778 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:14.740572    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:19:15.265527 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:15.265491    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 17 22:19:15.557246 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:15.557212    2585 generic.go:356] "Generic (PLEG): container finished" podID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerID="e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a" exitCode=2
Sep 17 22:19:15.557474 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:15.557296    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" event={"ID":"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1","Type":"ContainerDied","Data":"e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a"}
Sep 17 22:19:19.990182 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:19.990156    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:19:20.123873 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.123777    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-proxy-tls\") pod \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") "
Sep 17 22:19:20.123873 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.123847    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-rbac-proxy-sar-config\") pod \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") "
Sep 17 22:19:20.123873 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.123877    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrgp5\" (UniqueName: \"kubernetes.io/projected/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-api-access-nrgp5\") pod \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") "
Sep 17 22:19:20.124153 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.123971    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-cabundle-cert\") pod \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") "
Sep 17 22:19:20.124153 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.124018    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kserve-provision-location\") pod \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\" (UID: \"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1\") "
Sep 17 22:19:20.124337 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.124306    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" (UID: "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:20.124379 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.124248    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" (UID: "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:20.124379 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.124354    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" (UID: "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:20.126161 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.126141    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" (UID: "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:19:20.126251 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.126236    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-api-access-nrgp5" (OuterVolumeSpecName: "kube-api-access-nrgp5") pod "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" (UID: "1bdaa498-b6fb-4912-a4f7-24ffa6e382c1"). InnerVolumeSpecName "kube-api-access-nrgp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:20.225482 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.225420    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrgp5\" (UniqueName: \"kubernetes.io/projected/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-api-access-nrgp5\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:19:20.225482 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.225474    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-cabundle-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:19:20.225482 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.225486    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kserve-provision-location\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:19:20.225482 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.225495    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:19:20.225754 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.225505    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:19:20.579138 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.579101    2585 generic.go:356] "Generic (PLEG): container finished" podID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerID="206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f" exitCode=0
Sep 17 22:19:20.579338 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.579168    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"
Sep 17 22:19:20.579338 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.579185    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" event={"ID":"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1","Type":"ContainerDied","Data":"206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f"}
Sep 17 22:19:20.579338 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.579265    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv" event={"ID":"1bdaa498-b6fb-4912-a4f7-24ffa6e382c1","Type":"ContainerDied","Data":"b92011f907a2e040ad8bcc2090f1f53d2d5d95e875367af8bf16d46a8bb95f0f"}
Sep 17 22:19:20.579338 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.579302    2585 scope.go:117] "RemoveContainer" containerID="e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a"
Sep 17 22:19:20.587808 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.587786    2585 scope.go:117] "RemoveContainer" containerID="206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f"
Sep 17 22:19:20.595658 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.595608    2585 scope.go:117] "RemoveContainer" containerID="661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29"
Sep 17 22:19:20.597286 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.597262    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"]
Sep 17 22:19:20.601376 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.601346    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-j7gbv"]
Sep 17 22:19:20.604184 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.604168    2585 scope.go:117] "RemoveContainer" containerID="e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a"
Sep 17 22:19:20.604543 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:19:20.604525    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a\": container with ID starting with e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a not found: ID does not exist" containerID="e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a"
Sep 17 22:19:20.604595 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.604552    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a"} err="failed to get container status \"e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a\": rpc error: code = NotFound desc = could not find container \"e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a\": container with ID starting with e654dd47155e6cb82574d6a9a1c917a2524fc08c433ea973da6428efbe4eea8a not found: ID does not exist"
Sep 17 22:19:20.604595 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.604573    2585 scope.go:117] "RemoveContainer" containerID="206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f"
Sep 17 22:19:20.604845 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:19:20.604827    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f\": container with ID starting with 206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f not found: ID does not exist" containerID="206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f"
Sep 17 22:19:20.604888 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.604851    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f"} err="failed to get container status \"206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f\": rpc error: code = NotFound desc = could not find container \"206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f\": container with ID starting with 206eb32a8e616b1c0a8b9dd6d1020566d82c47f3bb877ef5b7904e5d0baee41f not found: ID does not exist"
Sep 17 22:19:20.604888 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.604867    2585 scope.go:117] "RemoveContainer" containerID="661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29"
Sep 17 22:19:20.605108 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:19:20.605091    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29\": container with ID starting with 661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29 not found: ID does not exist" containerID="661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29"
Sep 17 22:19:20.605154 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:20.605114    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29"} err="failed to get container status \"661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29\": rpc error: code = NotFound desc = could not find container \"661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29\": container with ID starting with 661e3580a522cfe750f33fc0d45830d9dbfbabc3b4d7e3a023283310c1d0db29 not found: ID does not exist"
Sep 17 22:19:22.463722 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:22.463688    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" path="/var/lib/kubelet/pods/1bdaa498-b6fb-4912-a4f7-24ffa6e382c1/volumes"
Sep 17 22:19:28.348388 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348348    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"]
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348664    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kube-rbac-proxy"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348676    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kube-rbac-proxy"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348684    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="storage-initializer"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348691    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="storage-initializer"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348704    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348709    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348720    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="storage-initializer"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348725    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="storage-initializer"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348735    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348740    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348745    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kube-rbac-proxy"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348750    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kube-rbac-proxy"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348806    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kserve-container"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348811    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdaa498-b6fb-4912-a4f7-24ffa6e382c1" containerName="kube-rbac-proxy"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348819    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kube-rbac-proxy"
Sep 17 22:19:28.348889 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.348825    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5561da14-2f61-4586-817a-febf36d9bd7b" containerName="kserve-container"
Sep 17 22:19:28.353416 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.353396    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.355847 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.355818    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 22:19:28.356053 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.356036    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:19:28.356138 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.356116    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:19:28.356732 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.356710    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:19:28.356732 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.356728    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:19:28.356880 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.356742    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 22:19:28.356880 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.356710    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:19:28.363292 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.363264    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"]
Sep 17 22:19:28.382473 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.382417    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67318066-8565-4597-8ac9-a62ef1fb744c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.382682 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.382488    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.382682 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.382521    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6tm\" (UniqueName: \"kubernetes.io/projected/67318066-8565-4597-8ac9-a62ef1fb744c-kube-api-access-pm6tm\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.382682 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.382567    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.382682 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.382589    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67318066-8565-4597-8ac9-a62ef1fb744c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.483796 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.483753    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.483796 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.483798    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67318066-8565-4597-8ac9-a62ef1fb744c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.484077 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.483820    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67318066-8565-4597-8ac9-a62ef1fb744c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.484077 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.483842    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.484077 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:19:28.483927    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 22:19:28.484077 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:19:28.484010    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67318066-8565-4597-8ac9-a62ef1fb744c-proxy-tls podName:67318066-8565-4597-8ac9-a62ef1fb744c nodeName:}" failed. No retries permitted until 2026-09-17 22:19:28.983989865 +0000 UTC m=+1270.846243326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/67318066-8565-4597-8ac9-a62ef1fb744c-proxy-tls") pod "isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" (UID: "67318066-8565-4597-8ac9-a62ef1fb744c") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 22:19:28.484293 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.483921    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6tm\" (UniqueName: \"kubernetes.io/projected/67318066-8565-4597-8ac9-a62ef1fb744c-kube-api-access-pm6tm\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.484293 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.484228    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67318066-8565-4597-8ac9-a62ef1fb744c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.484600 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.484581    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.484667 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.484640    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.492566 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.492544    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6tm\" (UniqueName: \"kubernetes.io/projected/67318066-8565-4597-8ac9-a62ef1fb744c-kube-api-access-pm6tm\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.986442 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.986376    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67318066-8565-4597-8ac9-a62ef1fb744c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:28.989140 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:28.989113    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67318066-8565-4597-8ac9-a62ef1fb744c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:29.263676 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:29.263579    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:29.540756 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:29.540690    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"]
Sep 17 22:19:29.543230 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:19:29.543200    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67318066_8565_4597_8ac9_a62ef1fb744c.slice/crio-ff93e0ff262f99a0da5bfae1b5553b9db1d93bcf144a4f4c9f301821c363e415 WatchSource:0}: Error finding container ff93e0ff262f99a0da5bfae1b5553b9db1d93bcf144a4f4c9f301821c363e415: Status 404 returned error can't find the container with id ff93e0ff262f99a0da5bfae1b5553b9db1d93bcf144a4f4c9f301821c363e415
Sep 17 22:19:29.617384 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:29.617349    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" event={"ID":"67318066-8565-4597-8ac9-a62ef1fb744c","Type":"ContainerStarted","Data":"1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18"}
Sep 17 22:19:29.617572 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:29.617447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" event={"ID":"67318066-8565-4597-8ac9-a62ef1fb744c","Type":"ContainerStarted","Data":"ff93e0ff262f99a0da5bfae1b5553b9db1d93bcf144a4f4c9f301821c363e415"}
Sep 17 22:19:33.634747 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:33.634705    2585 generic.go:356] "Generic (PLEG): container finished" podID="67318066-8565-4597-8ac9-a62ef1fb744c" containerID="1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18" exitCode=0
Sep 17 22:19:33.635265 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:33.634783    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" event={"ID":"67318066-8565-4597-8ac9-a62ef1fb744c","Type":"ContainerDied","Data":"1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18"}
Sep 17 22:19:34.640993 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:34.640956    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" event={"ID":"67318066-8565-4597-8ac9-a62ef1fb744c","Type":"ContainerStarted","Data":"528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793"}
Sep 17 22:19:34.641397 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:34.641024    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" event={"ID":"67318066-8565-4597-8ac9-a62ef1fb744c","Type":"ContainerStarted","Data":"38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04"}
Sep 17 22:19:34.641397 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:34.641218    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:34.659741 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:34.659679    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podStartSLOduration=2.774728733 podStartE2EDuration="6.659658148s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.884929415s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:19:34.658050351 +0000 UTC m=+1276.520303828" watchObservedRunningTime="2026-09-17 22:19:34.659658148 +0000 UTC m=+1276.521911630"
Sep 17 22:19:35.645105 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:35.645067    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:35.646662 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:35.646630    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:19:36.649228 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:36.649177    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:19:41.653615 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:41.653586    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:19:41.654080 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:41.654057    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:19:51.654108 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:19:51.654072    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:20:01.654546 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:20:01.654498    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:20:11.654060 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:20:11.654021    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:20:21.654255 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:20:21.654215    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:20:31.654422 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:20:31.654338    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:20:41.654035 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:20:41.653994    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:20:43.459648 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:20:43.459608    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:20:53.460241 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:20:53.460211    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:22:28.678669 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:28.678588    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"]
Sep 17 22:22:28.679216 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:28.678866    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" containerID="cri-o://38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04" gracePeriod=30
Sep 17 22:22:28.679216 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:28.678964    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kube-rbac-proxy" containerID="cri-o://528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793" gracePeriod=30
Sep 17 22:22:29.328033 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:29.327990    2585 generic.go:356] "Generic (PLEG): container finished" podID="67318066-8565-4597-8ac9-a62ef1fb744c" containerID="528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793" exitCode=2
Sep 17 22:22:29.328207 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:29.328067    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" event={"ID":"67318066-8565-4597-8ac9-a62ef1fb744c","Type":"ContainerDied","Data":"528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793"}
Sep 17 22:22:31.649608 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:31.649565    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 22:22:33.459610 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:33.459567    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 22:22:34.025531 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.025501    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:22:34.132675 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.132566    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-kube-rbac-proxy-sar-config\") pod \"67318066-8565-4597-8ac9-a62ef1fb744c\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") "
Sep 17 22:22:34.132675 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.132635    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67318066-8565-4597-8ac9-a62ef1fb744c-kserve-provision-location\") pod \"67318066-8565-4597-8ac9-a62ef1fb744c\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") "
Sep 17 22:22:34.132675 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.132661    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67318066-8565-4597-8ac9-a62ef1fb744c-proxy-tls\") pod \"67318066-8565-4597-8ac9-a62ef1fb744c\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") "
Sep 17 22:22:34.132961 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.132720    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm6tm\" (UniqueName: \"kubernetes.io/projected/67318066-8565-4597-8ac9-a62ef1fb744c-kube-api-access-pm6tm\") pod \"67318066-8565-4597-8ac9-a62ef1fb744c\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") "
Sep 17 22:22:34.132961 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.132749    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-cabundle-cert\") pod \"67318066-8565-4597-8ac9-a62ef1fb744c\" (UID: \"67318066-8565-4597-8ac9-a62ef1fb744c\") "
Sep 17 22:22:34.133032 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.132967    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67318066-8565-4597-8ac9-a62ef1fb744c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67318066-8565-4597-8ac9-a62ef1fb744c" (UID: "67318066-8565-4597-8ac9-a62ef1fb744c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:22:34.133032 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.133005    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67318066-8565-4597-8ac9-a62ef1fb744c" (UID: "67318066-8565-4597-8ac9-a62ef1fb744c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:22:34.133203 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.133178    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67318066-8565-4597-8ac9-a62ef1fb744c" (UID: "67318066-8565-4597-8ac9-a62ef1fb744c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:22:34.135032 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.135009    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67318066-8565-4597-8ac9-a62ef1fb744c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67318066-8565-4597-8ac9-a62ef1fb744c" (UID: "67318066-8565-4597-8ac9-a62ef1fb744c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:22:34.135152 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.135126    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67318066-8565-4597-8ac9-a62ef1fb744c-kube-api-access-pm6tm" (OuterVolumeSpecName: "kube-api-access-pm6tm") pod "67318066-8565-4597-8ac9-a62ef1fb744c" (UID: "67318066-8565-4597-8ac9-a62ef1fb744c"). InnerVolumeSpecName "kube-api-access-pm6tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:22:34.234300 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.234259    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:22:34.234300 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.234290    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67318066-8565-4597-8ac9-a62ef1fb744c-kserve-provision-location\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:22:34.234300 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.234300    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67318066-8565-4597-8ac9-a62ef1fb744c-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:22:34.234300 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.234309    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm6tm\" (UniqueName: \"kubernetes.io/projected/67318066-8565-4597-8ac9-a62ef1fb744c-kube-api-access-pm6tm\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:22:34.234585 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.234319    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67318066-8565-4597-8ac9-a62ef1fb744c-cabundle-cert\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:22:34.349954 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.349920    2585 generic.go:356] "Generic (PLEG): container finished" podID="67318066-8565-4597-8ac9-a62ef1fb744c" containerID="38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04" exitCode=0
Sep 17 22:22:34.350133 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.349980    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"
Sep 17 22:22:34.350133 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.350008    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" event={"ID":"67318066-8565-4597-8ac9-a62ef1fb744c","Type":"ContainerDied","Data":"38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04"}
Sep 17 22:22:34.350133 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.350090    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5" event={"ID":"67318066-8565-4597-8ac9-a62ef1fb744c","Type":"ContainerDied","Data":"ff93e0ff262f99a0da5bfae1b5553b9db1d93bcf144a4f4c9f301821c363e415"}
Sep 17 22:22:34.350254 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.350133    2585 scope.go:117] "RemoveContainer" containerID="528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793"
Sep 17 22:22:34.358899 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.358880    2585 scope.go:117] "RemoveContainer" containerID="38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04"
Sep 17 22:22:34.366681 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.366659    2585 scope.go:117] "RemoveContainer" containerID="1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18"
Sep 17 22:22:34.371250 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.371223    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"]
Sep 17 22:22:34.375083 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.375061    2585 scope.go:117] "RemoveContainer" containerID="528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793"
Sep 17 22:22:34.375209 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.375187    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-5glt5"]
Sep 17 22:22:34.375461 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:22:34.375421    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793\": container with ID starting with 528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793 not found: ID does not exist" containerID="528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793"
Sep 17 22:22:34.375510 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.375477    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793"} err="failed to get container status \"528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793\": rpc error: code = NotFound desc = could not find container \"528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793\": container with ID starting with 528c478fba21f13256e42d1d5c8b6c9a1cc2a0af312c0a51c9c86da857602793 not found: ID does not exist"
Sep 17 22:22:34.375510 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.375502    2585 scope.go:117] "RemoveContainer" containerID="38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04"
Sep 17 22:22:34.375816 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:22:34.375799    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04\": container with ID starting with 38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04 not found: ID does not exist" containerID="38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04"
Sep 17 22:22:34.375862 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.375822    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04"} err="failed to get container status \"38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04\": rpc error: code = NotFound desc = could not find container \"38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04\": container with ID starting with 38ac1b9ff1dc7644f483d8c4de18150b986f606d37e2ca6c38973453e00dfe04 not found: ID does not exist"
Sep 17 22:22:34.375862 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.375839    2585 scope.go:117] "RemoveContainer" containerID="1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18"
Sep 17 22:22:34.376098 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:22:34.376081    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18\": container with ID starting with 1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18 not found: ID does not exist" containerID="1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18"
Sep 17 22:22:34.376143 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.376103    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18"} err="failed to get container status \"1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18\": rpc error: code = NotFound desc = could not find container \"1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18\": container with ID starting with 1991e03244c5a0ffcb7ad8c4b02e8b018d29bdd5645553eb093417c7ddacaa18 not found: ID does not exist"
Sep 17 22:22:34.463646 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:34.463565    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" path="/var/lib/kubelet/pods/67318066-8565-4597-8ac9-a62ef1fb744c/volumes"
Sep 17 22:22:42.812316 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.812277    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"]
Sep 17 22:22:42.812725 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.812564    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="storage-initializer"
Sep 17 22:22:42.812725 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.812575    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="storage-initializer"
Sep 17 22:22:42.812725 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.812586    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container"
Sep 17 22:22:42.812725 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.812591    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container"
Sep 17 22:22:42.812725 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.812610    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kube-rbac-proxy"
Sep 17 22:22:42.812725 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.812616    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kube-rbac-proxy"
Sep 17 22:22:42.812725 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.812664    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kserve-container"
Sep 17 22:22:42.812725 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.812671    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="67318066-8565-4597-8ac9-a62ef1fb744c" containerName="kube-rbac-proxy"
Sep 17 22:22:42.815558 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.815541    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:42.818074 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.818050    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:22:42.818211 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.818082    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:22:42.818934 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.818904    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:22:42.819052 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.818969    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 22:22:42.819052 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.818977    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 22:22:42.825566 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.825544    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"]
Sep 17 22:22:42.899391 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.899352    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:42.899613 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.899415    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5kfg\" (UniqueName: \"kubernetes.io/projected/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-api-access-r5kfg\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:42.899613 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.899478    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:43.000018 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:42.999976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5kfg\" (UniqueName: \"kubernetes.io/projected/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-api-access-r5kfg\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:43.000018 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:43.000017    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:43.000306 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:43.000062    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:43.000306 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:22:43.000206    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-serving-cert: secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 17 22:22:43.000306 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:22:43.000302    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-proxy-tls podName:30ff2cfa-f7a0-4a04-85a2-6c42a514352f nodeName:}" failed. No retries permitted until 2026-09-17 22:22:43.500279067 +0000 UTC m=+1465.362532548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-proxy-tls") pod "message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" (UID: "30ff2cfa-f7a0-4a04-85a2-6c42a514352f") : secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 17 22:22:43.000728 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:43.000709    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:43.008356 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:43.008327    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5kfg\" (UniqueName: \"kubernetes.io/projected/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-api-access-r5kfg\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:43.504308 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:43.504267    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:43.506949 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:43.506925    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-vg6nq\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:43.726399 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:43.726350    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:44.000477 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:44.000377    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"]
Sep 17 22:22:44.003053 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:22:44.003020    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ff2cfa_f7a0_4a04_85a2_6c42a514352f.slice/crio-33e81fd97ad97176a912302a43aac8e67a7afb9e7d1b1d4400780c8c16cc7de5 WatchSource:0}: Error finding container 33e81fd97ad97176a912302a43aac8e67a7afb9e7d1b1d4400780c8c16cc7de5: Status 404 returned error can't find the container with id 33e81fd97ad97176a912302a43aac8e67a7afb9e7d1b1d4400780c8c16cc7de5
Sep 17 22:22:44.396263 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:44.396163    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" event={"ID":"30ff2cfa-f7a0-4a04-85a2-6c42a514352f","Type":"ContainerStarted","Data":"cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b"}
Sep 17 22:22:44.396263 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:44.396238    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" event={"ID":"30ff2cfa-f7a0-4a04-85a2-6c42a514352f","Type":"ContainerStarted","Data":"e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2"}
Sep 17 22:22:44.396514 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:44.396272    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:44.396514 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:44.396298    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" event={"ID":"30ff2cfa-f7a0-4a04-85a2-6c42a514352f","Type":"ContainerStarted","Data":"33e81fd97ad97176a912302a43aac8e67a7afb9e7d1b1d4400780c8c16cc7de5"}
Sep 17 22:22:44.415683 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:44.415627    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" podStartSLOduration=2.415608132 podStartE2EDuration="2.415608132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:22:44.413789052 +0000 UTC m=+1466.276042540" watchObservedRunningTime="2026-09-17 22:22:44.415608132 +0000 UTC m=+1466.277861656"
Sep 17 22:22:45.400722 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:45.400685    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:45.402613 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:45.402589    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:22:52.415324 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:22:52.415289    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:23:20.456071 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:23:20.456039    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:23:20.458237 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:23:20.458214    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:24:27.822997 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:27.822962    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-vg6nq_30ff2cfa-f7a0-4a04-85a2-6c42a514352f/kserve-container/0.log"
Sep 17 22:24:28.088255 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.088159    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"]
Sep 17 22:24:28.088492 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.088441    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" podUID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerName="kserve-container" containerID="cri-o://e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2" gracePeriod=30
Sep 17 22:24:28.088654 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.088498    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" podUID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerName="kube-rbac-proxy" containerID="cri-o://cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b" gracePeriod=30
Sep 17 22:24:28.346015 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.345947    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:24:28.502882 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.502846    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5kfg\" (UniqueName: \"kubernetes.io/projected/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-api-access-r5kfg\") pod \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") "
Sep 17 22:24:28.502882 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.502885    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-proxy-tls\") pod \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") "
Sep 17 22:24:28.503099 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.502949    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-rbac-proxy-sar-config\") pod \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\" (UID: \"30ff2cfa-f7a0-4a04-85a2-6c42a514352f\") "
Sep 17 22:24:28.503405 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.503364    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30ff2cfa-f7a0-4a04-85a2-6c42a514352f" (UID: "30ff2cfa-f7a0-4a04-85a2-6c42a514352f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:28.505277 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.505248    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30ff2cfa-f7a0-4a04-85a2-6c42a514352f" (UID: "30ff2cfa-f7a0-4a04-85a2-6c42a514352f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:24:28.505397 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.505269    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-api-access-r5kfg" (OuterVolumeSpecName: "kube-api-access-r5kfg") pod "30ff2cfa-f7a0-4a04-85a2-6c42a514352f" (UID: "30ff2cfa-f7a0-4a04-85a2-6c42a514352f"). InnerVolumeSpecName "kube-api-access-r5kfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:24:28.603885 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.603792    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:24:28.603885 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.603825    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5kfg\" (UniqueName: \"kubernetes.io/projected/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-kube-api-access-r5kfg\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:24:28.603885 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.603845    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30ff2cfa-f7a0-4a04-85a2-6c42a514352f-proxy-tls\") on node \"ip-10-0-137-232.ec2.internal\" DevicePath \"\""
Sep 17 22:24:28.812612 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.812570    2585 generic.go:356] "Generic (PLEG): container finished" podID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerID="cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b" exitCode=2
Sep 17 22:24:28.812612 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.812605    2585 generic.go:356] "Generic (PLEG): container finished" podID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerID="e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2" exitCode=2
Sep 17 22:24:28.812868 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.812632    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"
Sep 17 22:24:28.812868 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.812665    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" event={"ID":"30ff2cfa-f7a0-4a04-85a2-6c42a514352f","Type":"ContainerDied","Data":"cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b"}
Sep 17 22:24:28.812868 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.812742    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" event={"ID":"30ff2cfa-f7a0-4a04-85a2-6c42a514352f","Type":"ContainerDied","Data":"e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2"}
Sep 17 22:24:28.812868 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.812778    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq" event={"ID":"30ff2cfa-f7a0-4a04-85a2-6c42a514352f","Type":"ContainerDied","Data":"33e81fd97ad97176a912302a43aac8e67a7afb9e7d1b1d4400780c8c16cc7de5"}
Sep 17 22:24:28.812868 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.812810    2585 scope.go:117] "RemoveContainer" containerID="cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b"
Sep 17 22:24:28.821876 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.821856    2585 scope.go:117] "RemoveContainer" containerID="e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2"
Sep 17 22:24:28.831927 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.831897    2585 scope.go:117] "RemoveContainer" containerID="cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b"
Sep 17 22:24:28.832491 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:24:28.832464    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b\": container with ID starting with cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b not found: ID does not exist" containerID="cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b"
Sep 17 22:24:28.832675 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.832516    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b"} err="failed to get container status \"cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b\": rpc error: code = NotFound desc = could not find container \"cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b\": container with ID starting with cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b not found: ID does not exist"
Sep 17 22:24:28.832675 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.832544    2585 scope.go:117] "RemoveContainer" containerID="e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2"
Sep 17 22:24:28.833189 ip-10-0-137-232 kubenswrapper[2585]: E0917 22:24:28.833161    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2\": container with ID starting with e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2 not found: ID does not exist" containerID="e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2"
Sep 17 22:24:28.833307 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.833196    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2"} err="failed to get container status \"e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2\": rpc error: code = NotFound desc = could not find container \"e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2\": container with ID starting with e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2 not found: ID does not exist"
Sep 17 22:24:28.833307 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.833218    2585 scope.go:117] "RemoveContainer" containerID="cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b"
Sep 17 22:24:28.833543 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.833525    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b"} err="failed to get container status \"cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b\": rpc error: code = NotFound desc = could not find container \"cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b\": container with ID starting with cab489766479fd7f2310233b81dabf1fa308a8978db209872d24449e02311d1b not found: ID does not exist"
Sep 17 22:24:28.833603 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.833543    2585 scope.go:117] "RemoveContainer" containerID="e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2"
Sep 17 22:24:28.833773 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.833755    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2"} err="failed to get container status \"e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2\": rpc error: code = NotFound desc = could not find container \"e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2\": container with ID starting with e8e89690f4e9d6b73dca04e7d0201935d727177eb38f055ed733309aa096fab2 not found: ID does not exist"
Sep 17 22:24:28.834257 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.834239    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"]
Sep 17 22:24:28.838650 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:28.838624    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-vg6nq"]
Sep 17 22:24:30.463807 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:30.463768    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" path="/var/lib/kubelet/pods/30ff2cfa-f7a0-4a04-85a2-6c42a514352f/volumes"
Sep 17 22:24:59.867316 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.867272    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fcd62/must-gather-vj2jr"]
Sep 17 22:24:59.867975 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.867716    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerName="kserve-container"
Sep 17 22:24:59.867975 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.867734    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerName="kserve-container"
Sep 17 22:24:59.867975 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.867803    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerName="kube-rbac-proxy"
Sep 17 22:24:59.867975 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.867813    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerName="kube-rbac-proxy"
Sep 17 22:24:59.867975 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.867896    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerName="kube-rbac-proxy"
Sep 17 22:24:59.867975 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.867908    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ff2cfa-f7a0-4a04-85a2-6c42a514352f" containerName="kserve-container"
Sep 17 22:24:59.872085 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.872058    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcd62/must-gather-vj2jr"
Sep 17 22:24:59.875027 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.874991    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcd62\"/\"kube-root-ca.crt\""
Sep 17 22:24:59.875027 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.874991    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcd62\"/\"openshift-service-ca.crt\""
Sep 17 22:24:59.875785 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.875759    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fcd62\"/\"default-dockercfg-842mj\""
Sep 17 22:24:59.878480 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.878419    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcd62/must-gather-vj2jr"]
Sep 17 22:24:59.938116 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.938079    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgn9\" (UniqueName: \"kubernetes.io/projected/d574f16c-ffca-42db-9a19-6b6f0de86797-kube-api-access-fkgn9\") pod \"must-gather-vj2jr\" (UID: \"d574f16c-ffca-42db-9a19-6b6f0de86797\") " pod="openshift-must-gather-fcd62/must-gather-vj2jr"
Sep 17 22:24:59.938301 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:24:59.938159    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d574f16c-ffca-42db-9a19-6b6f0de86797-must-gather-output\") pod \"must-gather-vj2jr\" (UID: \"d574f16c-ffca-42db-9a19-6b6f0de86797\") " pod="openshift-must-gather-fcd62/must-gather-vj2jr"
Sep 17 22:25:00.039496 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:00.039449    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgn9\" (UniqueName: \"kubernetes.io/projected/d574f16c-ffca-42db-9a19-6b6f0de86797-kube-api-access-fkgn9\") pod \"must-gather-vj2jr\" (UID: \"d574f16c-ffca-42db-9a19-6b6f0de86797\") " pod="openshift-must-gather-fcd62/must-gather-vj2jr"
Sep 17 22:25:00.039695 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:00.039548    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d574f16c-ffca-42db-9a19-6b6f0de86797-must-gather-output\") pod \"must-gather-vj2jr\" (UID: \"d574f16c-ffca-42db-9a19-6b6f0de86797\") " pod="openshift-must-gather-fcd62/must-gather-vj2jr"
Sep 17 22:25:00.039858 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:00.039841    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d574f16c-ffca-42db-9a19-6b6f0de86797-must-gather-output\") pod \"must-gather-vj2jr\" (UID: \"d574f16c-ffca-42db-9a19-6b6f0de86797\") " pod="openshift-must-gather-fcd62/must-gather-vj2jr"
Sep 17 22:25:00.047122 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:00.047094    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgn9\" (UniqueName: \"kubernetes.io/projected/d574f16c-ffca-42db-9a19-6b6f0de86797-kube-api-access-fkgn9\") pod \"must-gather-vj2jr\" (UID: \"d574f16c-ffca-42db-9a19-6b6f0de86797\") " pod="openshift-must-gather-fcd62/must-gather-vj2jr"
Sep 17 22:25:00.186127 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:00.186036    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcd62/must-gather-vj2jr"
Sep 17 22:25:00.456505 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:00.456273    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcd62/must-gather-vj2jr"]
Sep 17 22:25:00.459179 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:25:00.459148    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd574f16c_ffca_42db_9a19_6b6f0de86797.slice/crio-9cd0d3345fd9a2f6f61bf1f156a4d75aca1563964bc0b42b0ef2aebab0310f39 WatchSource:0}: Error finding container 9cd0d3345fd9a2f6f61bf1f156a4d75aca1563964bc0b42b0ef2aebab0310f39: Status 404 returned error can't find the container with id 9cd0d3345fd9a2f6f61bf1f156a4d75aca1563964bc0b42b0ef2aebab0310f39
Sep 17 22:25:00.461272 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:00.461254    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:25:00.936155 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:00.936103    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcd62/must-gather-vj2jr" event={"ID":"d574f16c-ffca-42db-9a19-6b6f0de86797","Type":"ContainerStarted","Data":"9cd0d3345fd9a2f6f61bf1f156a4d75aca1563964bc0b42b0ef2aebab0310f39"}
Sep 17 22:25:01.944420 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:01.944384    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcd62/must-gather-vj2jr" event={"ID":"d574f16c-ffca-42db-9a19-6b6f0de86797","Type":"ContainerStarted","Data":"cf0f9b37e8ce2d6ce5b60b189ca4b8bac2735db61a50e1770f7a8319deea2bec"}
Sep 17 22:25:01.944926 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:01.944451    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcd62/must-gather-vj2jr" event={"ID":"d574f16c-ffca-42db-9a19-6b6f0de86797","Type":"ContainerStarted","Data":"4f1feb077cabb356ad7ee4a8f7b9a18514ce5c406bdaa54dc8ab42a8293e77f2"}
Sep 17 22:25:01.960378 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:01.960319    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fcd62/must-gather-vj2jr" podStartSLOduration=2.092387716 podStartE2EDuration="2.960297601s" totalImagesPullingTime="867.909885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:25:01.958901243 +0000 UTC m=+1603.821154720" watchObservedRunningTime="2026-09-17 22:25:01.960297601 +0000 UTC m=+1603.822551078"
Sep 17 22:25:02.753382 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:02.753352    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tr566_45d76677-9235-42a6-8963-04df258d406a/global-pull-secret-syncer/0.log"
Sep 17 22:25:02.863394 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:02.863263    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rfrx9_322caaa1-110c-4e1a-96cf-a3d4091e9b56/konnectivity-agent/0.log"
Sep 17 22:25:02.907357 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:02.907325    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-232.ec2.internal_dc9ce5716ceb5cf8fa3712948819d326/haproxy/0.log"
Sep 17 22:25:07.551843 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:07.551789    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-qdd9f_43882037-9994-4040-a7cd-bc334661ace1/cluster-monitoring-operator/0.log"
Sep 17 22:25:07.750229 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:07.750140    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvtl6_6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2/node-exporter/0.log"
Sep 17 22:25:07.778320 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:07.778288    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvtl6_6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2/kube-rbac-proxy/0.log"
Sep 17 22:25:07.802048 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:07.801965    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvtl6_6c529f35-bbf5-4ecb-b5fc-4f43a15f07e2/init-textfile/0.log"
Sep 17 22:25:11.111092 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.111055    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"]
Sep 17 22:25:11.117506 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.117470    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.123713 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.123679    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"]
Sep 17 22:25:11.262827 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.262785    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-podres\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.263103 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.263084    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-lib-modules\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.263232 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.263219    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-sys\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.263346 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.263332    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5672\" (UniqueName: \"kubernetes.io/projected/d7a53b91-d1b1-4e74-a5f3-3110537f3415-kube-api-access-w5672\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.263480 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.263466    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-proc\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.364778 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.364689    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-proc\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.364778 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.364759    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-podres\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.364999 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.364796    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-lib-modules\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.364999 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.364832    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-sys\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.364999 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.364840    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-proc\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.364999 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.364884    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5672\" (UniqueName: \"kubernetes.io/projected/d7a53b91-d1b1-4e74-a5f3-3110537f3415-kube-api-access-w5672\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.364999 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.364920    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-sys\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.364999 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.364950    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-podres\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.364999 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.364972    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7a53b91-d1b1-4e74-a5f3-3110537f3415-lib-modules\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.375321 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.375281    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5672\" (UniqueName: \"kubernetes.io/projected/d7a53b91-d1b1-4e74-a5f3-3110537f3415-kube-api-access-w5672\") pod \"perf-node-gather-daemonset-tgvp2\" (UID: \"d7a53b91-d1b1-4e74-a5f3-3110537f3415\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.431460 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.431398    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-5rcm6_bdf61d98-cf89-4bae-b903-48689855cba0/volume-data-source-validator/0.log"
Sep 17 22:25:11.435060 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.435024    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:11.736917 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.736885    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"]
Sep 17 22:25:11.740050 ip-10-0-137-232 kubenswrapper[2585]: W0917 22:25:11.740020    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd7a53b91_d1b1_4e74_a5f3_3110537f3415.slice/crio-52a933ce8b4fc772f7fee3eeaf03129ccfa0d0561f5967fd49c3f7b3f441dda3 WatchSource:0}: Error finding container 52a933ce8b4fc772f7fee3eeaf03129ccfa0d0561f5967fd49c3f7b3f441dda3: Status 404 returned error can't find the container with id 52a933ce8b4fc772f7fee3eeaf03129ccfa0d0561f5967fd49c3f7b3f441dda3
Sep 17 22:25:11.999525 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.999442    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2" event={"ID":"d7a53b91-d1b1-4e74-a5f3-3110537f3415","Type":"ContainerStarted","Data":"890406b889ab826bff1bf30fda4b4fa1112413a1d13ebb23cefabcf32844c2a8"}
Sep 17 22:25:11.999525 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.999496    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2" event={"ID":"d7a53b91-d1b1-4e74-a5f3-3110537f3415","Type":"ContainerStarted","Data":"52a933ce8b4fc772f7fee3eeaf03129ccfa0d0561f5967fd49c3f7b3f441dda3"}
Sep 17 22:25:11.999711 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:11.999585    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:12.017627 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:12.017553    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2" podStartSLOduration=1.017528457 podStartE2EDuration="1.017528457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:25:12.016062055 +0000 UTC m=+1613.878315531" watchObservedRunningTime="2026-09-17 22:25:12.017528457 +0000 UTC m=+1613.879781935"
Sep 17 22:25:12.466290 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:12.466264    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5rqh_a0157d6a-2e6f-4af0-88ad-842ddc9003ed/dns/0.log"
Sep 17 22:25:12.490301 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:12.490271    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5rqh_a0157d6a-2e6f-4af0-88ad-842ddc9003ed/kube-rbac-proxy/0.log"
Sep 17 22:25:12.541796 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:12.541735    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9rxzt_cdf9f479-ec36-4720-b83f-a0484c14cda4/dns-node-resolver/0.log"
Sep 17 22:25:13.013337 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:13.013302    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54f647d6c4-6hdhd_58690a93-436a-45f3-ab67-16d575d3d6b9/registry/0.log"
Sep 17 22:25:13.058903 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:13.058852    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jbrr7_ac293f58-ae69-4c22-8804-44545908c47c/node-ca/0.log"
Sep 17 22:25:14.190140 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:14.190104    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zlf9g_269460ba-d873-42ce-a6c0-e6b4d64fc627/serve-healthcheck-canary/0.log"
Sep 17 22:25:14.626602 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:14.626559    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h8ccm_c540354a-ac88-46ed-8a86-bec4fbb8c082/kube-rbac-proxy/0.log"
Sep 17 22:25:14.648686 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:14.648658    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h8ccm_c540354a-ac88-46ed-8a86-bec4fbb8c082/exporter/0.log"
Sep 17 22:25:14.670850 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:14.670821    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h8ccm_c540354a-ac88-46ed-8a86-bec4fbb8c082/extractor/0.log"
Sep 17 22:25:17.044291 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:17.044262    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-bjj8m_e2beca00-626f-4e8a-849e-937f9a984152/server/0.log"
Sep 17 22:25:17.198473 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:17.198439    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-prmkq_0e8ef02a-a3ff-4091-bdf3-4b2605013977/manager/0.log"
Sep 17 22:25:17.288957 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:17.288931    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-t79z2_0ed0b33c-6cf3-476b-b263-852224b92a2f/seaweedfs/0.log"
Sep 17 22:25:18.024230 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:18.024199    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-tgvp2"
Sep 17 22:25:21.863853 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:21.863816    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-mh7vm_7b514b24-b573-493e-bd63-ff52b7f4a787/kube-storage-version-migrator-operator/1.log"
Sep 17 22:25:21.865045 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:21.865017    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-mh7vm_7b514b24-b573-493e-bd63-ff52b7f4a787/kube-storage-version-migrator-operator/0.log"
Sep 17 22:25:22.898683 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:22.898607    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5lmb6_68f7b17b-4a7c-4f10-99f0-84652c10602d/kube-multus/0.log"
Sep 17 22:25:23.087667 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:23.087637    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qcl8l_5116c2df-4d3f-4892-91a3-b04ffe887787/kube-multus-additional-cni-plugins/0.log"
Sep 17 22:25:23.108370 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:23.108325    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qcl8l_5116c2df-4d3f-4892-91a3-b04ffe887787/egress-router-binary-copy/0.log"
Sep 17 22:25:23.128152 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:23.128102    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qcl8l_5116c2df-4d3f-4892-91a3-b04ffe887787/cni-plugins/0.log"
Sep 17 22:25:23.148745 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:23.148672    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qcl8l_5116c2df-4d3f-4892-91a3-b04ffe887787/bond-cni-plugin/0.log"
Sep 17 22:25:23.169649 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:23.169595    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qcl8l_5116c2df-4d3f-4892-91a3-b04ffe887787/routeoverride-cni/0.log"
Sep 17 22:25:23.190670 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:23.190618    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qcl8l_5116c2df-4d3f-4892-91a3-b04ffe887787/whereabouts-cni-bincopy/0.log"
Sep 17 22:25:23.210886 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:23.210854    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qcl8l_5116c2df-4d3f-4892-91a3-b04ffe887787/whereabouts-cni/0.log"
Sep 17 22:25:23.420750 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:23.420657    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lm5n6_12a3a05c-bf13-4011-a95e-36271b66ec5d/network-metrics-daemon/0.log"
Sep 17 22:25:23.441087 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:23.441057    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lm5n6_12a3a05c-bf13-4011-a95e-36271b66ec5d/kube-rbac-proxy/0.log"
Sep 17 22:25:24.530343 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:24.530297    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-controller/0.log"
Sep 17 22:25:24.552517 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:24.552491    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/1.log"
Sep 17 22:25:24.552517 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:24.552503    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovn-acl-logging/0.log"
Sep 17 22:25:24.574679 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:24.574651    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/kube-rbac-proxy-node/0.log"
Sep 17 22:25:24.602841 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:24.602807    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 22:25:24.628079 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:24.628045    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/northd/0.log"
Sep 17 22:25:24.649668 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:24.649638    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/nbdb/0.log"
Sep 17 22:25:24.673270 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:24.673245    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/sbdb/0.log"
Sep 17 22:25:24.721457 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:24.721087    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9ddd_9ec01088-cc90-4188-a393-e872a5f5be60/ovnkube-controller/0.log"
Sep 17 22:25:25.888833 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:25.888800    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-jqwrl_af9f97fa-dccd-4256-bd7d-a1ac2d005eb8/check-endpoints/0.log"
Sep 17 22:25:25.912081 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:25.912053    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-m2r9c_e3dbb9bd-4956-4a4f-baf5-0fdfe8ec4dfb/network-check-target-container/0.log"
Sep 17 22:25:26.856501 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:26.856464    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-f64cd_50e0dc15-2515-4770-8887-9643beec1691/iptables-alerter/0.log"
Sep 17 22:25:27.587867 ip-10-0-137-232 kubenswrapper[2585]: I0917 22:25:27.587834    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-j79q4_40867f68-4f1a-49e5-a580-b872ac068cac/tuned/0.log"