Sep 17 17:33:43.560979 ip-10-0-135-179 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:33:43.987548 ip-10-0-135-179 kubenswrapper[2521]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:43.987548 ip-10-0-135-179 kubenswrapper[2521]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:33:43.987548 ip-10-0-135-179 kubenswrapper[2521]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:43.987548 ip-10-0-135-179 kubenswrapper[2521]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:33:43.987548 ip-10-0-135-179 kubenswrapper[2521]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:43.990087 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.989951    2521 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:33:43.993446 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993424    2521 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:43.993446 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993441    2521 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:43.993446 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993446    2521 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:43.993446 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993450    2521 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:43.993446 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993454    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993470    2521 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993474    2521 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993478    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993483    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993487    2521 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993491    2521 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993495    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993499    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993502    2521 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993506    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993511    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993515    2521 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993519    2521 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993526    2521 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993531    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993535    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993539    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993543    2521 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993547    2521 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:43.993732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993551    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993555    2521 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993559    2521 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993563    2521 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993568    2521 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993572    2521 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993576    2521 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993580    2521 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993585    2521 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993589    2521 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993593    2521 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993597    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993601    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993605    2521 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993613    2521 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993618    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993623    2521 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993628    2521 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993633    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993637    2521 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:43.994651 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993641    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993645    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993649    2521 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993654    2521 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993658    2521 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993662    2521 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993667    2521 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993671    2521 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993676    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993680    2521 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993685    2521 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993689    2521 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993693    2521 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993697    2521 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993702    2521 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993706    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993729    2521 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993734    2521 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993738    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:43.995537 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993743    2521 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993747    2521 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993756    2521 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993762    2521 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993767    2521 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993771    2521 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993775    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993780    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993786    2521 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993791    2521 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993795    2521 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993800    2521 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993804    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993808    2521 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993813    2521 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993817    2521 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993821    2521 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993825    2521 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993830    2521 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:43.996341 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993833    2521 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993838    2521 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993842    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993846    2521 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993850    2521 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993854    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993859    2521 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993863    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993867    2521 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993871    2521 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993875    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993882    2521 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993888    2521 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993892    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993897    2521 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993901    2521 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.993905    2521 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995091    2521 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995104    2521 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:43.997173 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995110    2521 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995116    2521 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995124    2521 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995131    2521 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995136    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995141    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995145    2521 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995151    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995155    2521 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995161    2521 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995165    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995170    2521 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995174    2521 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995179    2521 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995184    2521 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995188    2521 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995193    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995198    2521 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995202    2521 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:43.998000 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995208    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995212    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995217    2521 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995222    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995226    2521 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995231    2521 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995236    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995241    2521 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995245    2521 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995250    2521 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995254    2521 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995258    2521 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995262    2521 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995266    2521 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995271    2521 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995276    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995280    2521 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995285    2521 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995290    2521 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995294    2521 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995298    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:43.998681 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995302    2521 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995306    2521 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995310    2521 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995314    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995318    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995323    2521 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995330    2521 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995336    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995341    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995346    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995351    2521 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995355    2521 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995360    2521 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995364    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995368    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995372    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995377    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995381    2521 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995385    2521 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:43.999271 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995389    2521 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995393    2521 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995398    2521 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995402    2521 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995406    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995410    2521 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995415    2521 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995421    2521 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995426    2521 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995430    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995435    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995440    2521 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995444    2521 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995448    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995452    2521 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995457    2521 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995461    2521 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995465    2521 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995470    2521 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995474    2521 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:43.999755 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995478    2521 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995482    2521 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995486    2521 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995491    2521 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995495    2521 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995499    2521 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995503    2521 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995508    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995512    2521 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995516    2521 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995520    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995524    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995528    2521 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995532    2521 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995537    2521 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995541    2521 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995545    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.995549    2521 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996156    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.000462 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996165    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996169    2521 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996174    2521 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996179    2521 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996183    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996188    2521 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996193    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996197    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996202    2521 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996206    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996210    2521 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996214    2521 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996218    2521 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996223    2521 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996227    2521 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996232    2521 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996237    2521 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996241    2521 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996245    2521 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996249    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.000953 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996254    2521 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996258    2521 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996262    2521 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996266    2521 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996271    2521 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996275    2521 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996279    2521 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996284    2521 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996288    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996293    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996297    2521 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996301    2521 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996305    2521 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996309    2521 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996313    2521 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996318    2521 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996323    2521 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996327    2521 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996331    2521 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996335    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.001465 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996339    2521 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996346    2521 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996351    2521 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996356    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996360    2521 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996364    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996368    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996373    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996377    2521 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996382    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996386    2521 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996391    2521 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996395    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996400    2521 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996404    2521 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996408    2521 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996412    2521 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996416    2521 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996422    2521 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.002070 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996426    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996431    2521 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996435    2521 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996440    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996444    2521 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996448    2521 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996452    2521 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996456    2521 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996460    2521 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996465    2521 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996470    2521 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996474    2521 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996479    2521 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996483    2521 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996488    2521 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996493    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996497    2521 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996501    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996505    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996510    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.002533 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996515    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996519    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996525    2521 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996531    2521 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996538    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996543    2521 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996552    2521 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996556    2521 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996561    2521 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996566    2521 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996570    2521 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996574    2521 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996578    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996582    2521 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996586    2521 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996591    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996595    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996599    2521 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.996604    2521 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996729    2521 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:33:44.003088 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996776    2521 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996785    2521 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996792    2521 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996801    2521 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996806    2521 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996813    2521 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996820    2521 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996825    2521 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996829    2521 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996835    2521 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996840    2521 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996846    2521 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996850    2521 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996855    2521 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996863    2521 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996867    2521 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996872    2521 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996880    2521 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996887    2521 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996892    2521 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996897    2521 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996903    2521 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996909    2521 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:33:44.003585 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996914    2521 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996919    2521 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996924    2521 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996930    2521 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996935    2521 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996939    2521 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996944    2521 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996949    2521 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996955    2521 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996960    2521 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996965    2521 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996970    2521 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996975    2521 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996981    2521 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996987    2521 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996992    2521 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.996997    2521 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997001    2521 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997006    2521 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997013    2521 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997018    2521 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997022    2521 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997027    2521 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997034    2521 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997039    2521 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:33:44.004202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997044    2521 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997048    2521 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997054    2521 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997059    2521 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997064    2521 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997070    2521 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997075    2521 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997080    2521 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997085    2521 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997090    2521 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997095    2521 flags.go:64] FLAG: --help="false"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997100    2521 flags.go:64] FLAG: --hostname-override="ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997105    2521 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997109    2521 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997114    2521 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997120    2521 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997126    2521 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997131    2521 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997136    2521 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997140    2521 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997145    2521 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997151    2521 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997157    2521 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997161    2521 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:33:44.004840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997166    2521 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997171    2521 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997175    2521 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997180    2521 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997184    2521 flags.go:64] FLAG: --lock-file=""
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997190    2521 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997197    2521 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997202    2521 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997211    2521 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997215    2521 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997222    2521 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997226    2521 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997231    2521 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997237    2521 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997241    2521 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997246    2521 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997253    2521 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997258    2521 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997264    2521 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997269    2521 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997274    2521 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997279    2521 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997283    2521 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997288    2521 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997293    2521 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:33:44.005430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997298    2521 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997308    2521 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997313    2521 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997318    2521 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997323    2521 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997328    2521 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997336    2521 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997342    2521 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997347    2521 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997352    2521 flags.go:64] FLAG: --port="10250"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997357    2521 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997361    2521 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-004ede3b3c71452b6"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997366    2521 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997371    2521 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997378    2521 flags.go:64] FLAG: --register-node="true"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997383    2521 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997388    2521 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997395    2521 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997399    2521 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997404    2521 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997410    2521 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997415    2521 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997420    2521 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997424    2521 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997429    2521 flags.go:64] FLAG: --runonce="false"
Sep 17 17:33:44.006058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997434    2521 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997439    2521 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997443    2521 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997448    2521 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997453    2521 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997458    2521 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997463    2521 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997468    2521 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997473    2521 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997478    2521 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997482    2521 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997487    2521 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997492    2521 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997497    2521 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997502    2521 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997511    2521 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997516    2521 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997520    2521 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997531    2521 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997536    2521 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997541    2521 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997548    2521 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997553    2521 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997557    2521 flags.go:64] FLAG: --v="2"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997565    2521 flags.go:64] FLAG: --version="false"
Sep 17 17:33:44.006677 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997571    2521 flags.go:64] FLAG: --vmodule=""
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997578    2521 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.997583    2521 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997792    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997800    2521 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997806    2521 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997811    2521 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997817    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997821    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997826    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997830    2521 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997834    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997838    2521 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997842    2521 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997846    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997851    2521 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997855    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997860    2521 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997864    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997868    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.007346 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997872    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997877    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997882    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997887    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997891    2521 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997895    2521 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997900    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997904    2521 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997910    2521 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997914    2521 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997919    2521 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997927    2521 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997933    2521 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997938    2521 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997942    2521 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997947    2521 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997952    2521 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997956    2521 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997960    2521 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.007931 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997965    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997969    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997974    2521 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997978    2521 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997982    2521 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997987    2521 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997991    2521 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997995    2521 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.997999    2521 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998003    2521 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998007    2521 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998011    2521 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998015    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998020    2521 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998024    2521 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998028    2521 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998033    2521 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998037    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998042    2521 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998046    2521 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.008397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998050    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998056    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998061    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998065    2521 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998071    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998075    2521 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998080    2521 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998084    2521 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998088    2521 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998092    2521 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998096    2521 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998100    2521 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998105    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998109    2521 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998113    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998117    2521 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998122    2521 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998126    2521 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998130    2521 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998134    2521 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.008902 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998139    2521 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998143    2521 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998147    2521 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998151    2521 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998157    2521 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998163    2521 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998168    2521 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998173    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998178    2521 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998184    2521 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998189    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998194    2521 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998198    2521 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998204    2521 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998208    2521 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998212    2521 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998217    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998222    2521 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998226    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.009411 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998230    2521 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998234    2521 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998239    2521 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:43.998244    2521 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:43.998253    2521 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.006341    2521 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.006356    2521 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.006374    2521 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.006379    2521 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006435    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006440    2521 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006443    2521 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006446    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006449    2521 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006451    2521 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.009886 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006454    2521 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006458    2521 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006461    2521 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006463    2521 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006466    2521 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006469    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006471    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006474    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006477    2521 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006479    2521 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006482    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006485    2521 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006487    2521 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006490    2521 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006492    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006495    2521 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006497    2521 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006501    2521 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006505    2521 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.010249 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006508    2521 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006511    2521 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006514    2521 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006517    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006523    2521 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006526    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006530    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006533    2521 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006535    2521 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006538    2521 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006541    2521 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006544    2521 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006547    2521 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006549    2521 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006552    2521 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006555    2521 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006557    2521 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006560    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006562    2521 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006565    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.010764 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006568    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006570    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006573    2521 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006576    2521 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006578    2521 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006581    2521 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006584    2521 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006586    2521 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006589    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006591    2521 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006594    2521 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006597    2521 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006600    2521 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006602    2521 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006605    2521 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006608    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006611    2521 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006614    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006618    2521 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.011237 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006622    2521 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006626    2521 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006629    2521 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006632    2521 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006635    2521 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006638    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006641    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006644    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006646    2521 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006649    2521 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006651    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006654    2521 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006656    2521 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006659    2521 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006661    2521 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006664    2521 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006667    2521 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006669    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006672    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006674    2521 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.011695 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006677    2521 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006680    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006683    2521 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006685    2521 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006688    2521 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006691    2521 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006694    2521 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006697    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006700    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006726    2521 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006730    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006733    2521 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006736    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006739    2521 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.012191 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006741    2521 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.006747    2521 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006870    2521 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006875    2521 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006878    2521 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006881    2521 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006884    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006887    2521 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006889    2521 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006892    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006895    2521 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006898    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006900    2521 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006903    2521 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006906    2521 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:44.012527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006908    2521 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006911    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006913    2521 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006916    2521 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006919    2521 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006921    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006924    2521 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006926    2521 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006929    2521 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006932    2521 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006935    2521 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006938    2521 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006940    2521 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006943    2521 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006945    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006948    2521 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006951    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006953    2521 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006956    2521 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006959    2521 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:44.012906 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006961    2521 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006964    2521 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006966    2521 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006969    2521 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006971    2521 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006973    2521 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006976    2521 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006978    2521 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006981    2521 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006984    2521 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006986    2521 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006988    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006991    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006994    2521 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006996    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.006999    2521 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007002    2521 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007004    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007007    2521 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007009    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:44.013397 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007012    2521 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007015    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007017    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007020    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007022    2521 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007025    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007027    2521 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007030    2521 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007033    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007035    2521 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007038    2521 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007040    2521 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007043    2521 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007046    2521 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007048    2521 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007051    2521 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007053    2521 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007056    2521 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007058    2521 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007061    2521 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:44.013992 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007063    2521 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007066    2521 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007068    2521 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007071    2521 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007074    2521 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007077    2521 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007080    2521 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007085    2521 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007088    2521 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007091    2521 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007094    2521 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007097    2521 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007099    2521 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007103    2521 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007106    2521 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007108    2521 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007111    2521 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007114    2521 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007116    2521 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:44.014472 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007119    2521 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007121    2521 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007126    2521 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007129    2521 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007132    2521 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007134    2521 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:44.007137    2521 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.007142    2521 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.007244    2521 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.009629    2521 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.010745    2521 server.go:1019] "Starting client certificate rotation"
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.010846    2521 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.010879    2521 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:44.015062 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.013559    2521 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:44.017242 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.017219    2521 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:44.033038 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.033019    2521 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:33:44.034658 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.034644    2521 log.go:25] "Validated CRI v1 image API"
Sep 17 17:33:44.035539 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.035515    2521 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:33:44.039270 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.039253    2521 fs.go:135] Filesystem UUIDs: map[744b070a-9074-44f5-a518-02e4e55e478a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 abb010c6-0864-4d49-ba69-e84ebcf19a30:/dev/nvme0n1p4]
Sep 17 17:33:44.039325 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.039271    2521 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:33:44.044418 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.044313    2521 manager.go:216] Machine: {Timestamp:2026-09-17 17:33:44.042863918 +0000 UTC m=+0.374714041 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099361 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2490f9a7f5885de37bbf876b6eee4b SystemUUID:ec2490f9-a7f5-885d-e37b-bf876b6eee4b BootID:717afa5c-c438-4b90-a687-c49bff2bff60 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e8:c3:96:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e8:c3:96:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:ac:04:22:22:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:33:44.044418 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.044416    2521 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:33:44.044526 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.044491    2521 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:33:44.047376 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.047353    2521 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:33:44.047586 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.047382    2521 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-179.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:33:44.049110 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.049098    2521 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:33:44.049149 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.049113    2521 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:33:44.049149 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.049122    2521 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:44.049961 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.049950    2521 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:44.050032 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.050024    2521 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:33:44.054452 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.054438    2521 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:33:44.055223 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.055212    2521 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:44.055347 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.055338    2521 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:33:44.055376 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.055370    2521 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:33:44.055403 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.055383    2521 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:33:44.055403 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.055399    2521 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:33:44.055452 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.055408    2521 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:33:44.055452 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.055418    2521 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:33:44.055781 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.055770    2521 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:44.055831 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.055788    2521 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:44.058255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.058235    2521 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:33:44.061141 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.061127    2521 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:33:44.061190 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.061159    2521 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:33:44.062454 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062444    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:33:44.062491 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062460    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:33:44.062491 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062466    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:33:44.062491 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062471    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:33:44.062491 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062482    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:33:44.062491 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062490    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:33:44.062653 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062507    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:33:44.062653 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062512    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:33:44.062653 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062519    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:33:44.062653 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062525    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:33:44.062653 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062533    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:33:44.062653 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.062538    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:33:44.063467 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.063458    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:33:44.063500 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.063469    2521 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:33:44.068276 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.068260    2521 server.go:1268] "Started kubelet"
Sep 17 17:33:44.068352 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.068315    2521 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:33:44.068402 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.068343    2521 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:33:44.068434 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.068410    2521 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:33:44.068993 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.068972    2521 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-179.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:33:44.069200 ip-10-0-135-179 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:33:44.070233 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.069266    2521 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:33:44.070233 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.069561    2521 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-179.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:33:44.070913 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.070382    2521 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:33:44.070913 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.070641    2521 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:33:44.075522 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.073766    2521 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-179.ec2.internal.18d62c217107b479  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.ec2.internal,UID:ip-10-0-135-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.068224121 +0000 UTC m=+0.400074244,LastTimestamp:2026-09-17 17:33:44.068224121 +0000 UTC m=+0.400074244,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.ec2.internal,}"
Sep 17 17:33:44.075522 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.075364    2521 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:33:44.079158 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.079144    2521 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:33:44.079232 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.079158    2521 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:33:44.079282 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.079214    2521 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-179.ec2.internal\" not found"
Sep 17 17:33:44.080029 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.080011    2521 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:33:44.080280 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.080148    2521 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:33:44.080353 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.080284    2521 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:33:44.080401 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.080385    2521 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:33:44.080401 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.080393    2521 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:33:44.081587 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.081562    2521 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:33:44.081664 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.081617    2521 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-179.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:33:44.082588 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.082574    2521 factory.go:153] Registering CRI-O factory
Sep 17 17:33:44.082660 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.082607    2521 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:33:44.082660 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.082647    2521 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:33:44.082660 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.082654    2521 factory.go:55] Registering systemd factory
Sep 17 17:33:44.082660 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.082659    2521 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:33:44.082878 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.082679    2521 factory.go:103] Registering Raw factory
Sep 17 17:33:44.082878 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.082691    2521 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:33:44.083104 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.083089    2521 manager.go:349] Starting recovery of all containers
Sep 17 17:33:44.093299 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.093282    2521 manager.go:354] Recovery completed
Sep 17 17:33:44.100446 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.100431    2521 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:44.102419 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.102394    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:44.102490 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.102433    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:44.102490 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.102443    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:44.103134 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.103116    2521 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:33:44.103134 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.103133    2521 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:33:44.103277 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.103139    2521 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:44.103277 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.103183    2521 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:44.103821 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.103803    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:44.103886 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.103833    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:44.103886 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.103844    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:44.105376 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.105364    2521 policy_none.go:49] "None policy: Start"
Sep 17 17:33:44.105433 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.105379    2521 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:33:44.105433 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.105390    2521 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:33:44.107394 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.107384    2521 policy_none.go:47] "Start"
Sep 17 17:33:44.118511 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.118444    2521 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-179.ec2.internal.18d62c2173117c72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.ec2.internal,UID:ip-10-0-135-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-179.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.10241957 +0000 UTC m=+0.434269692,LastTimestamp:2026-09-17 17:33:44.10241957 +0000 UTC m=+0.434269692,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.ec2.internal,}"
Sep 17 17:33:44.121504 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.121428    2521 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-179.ec2.internal.18d62c217311c2a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.ec2.internal,UID:ip-10-0-135-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-179.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.102437538 +0000 UTC m=+0.434287660,LastTimestamp:2026-09-17 17:33:44.102437538 +0000 UTC m=+0.434287660,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.ec2.internal,}"
Sep 17 17:33:44.126482 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.126449    2521 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sz6bw"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.134608    2521 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-179.ec2.internal.18d62c217311eb7b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.ec2.internal,UID:ip-10-0-135-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-179.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.102447995 +0000 UTC m=+0.434298119,LastTimestamp:2026-09-17 17:33:44.102447995 +0000 UTC m=+0.434298119,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.ec2.internal,}"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.137947    2521 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sz6bw"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.142344    2521 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.142376    2521 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.142385    2521 server.go:87] "Starting device plugin registration server"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.142750    2521 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.142765    2521 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.142849    2521 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.142972    2521 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.142982    2521 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.143190    2521 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-179.ec2.internal.18d62c2173117c72\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-179.ec2.internal.18d62c2173117c72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.ec2.internal,UID:ip-10-0-135-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-179.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.10241957 +0000 UTC m=+0.434269692,LastTimestamp:2026-09-17 17:33:44.1038196 +0000 UTC m=+0.435669728,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.ec2.internal,}"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.143767    2521 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.143815    2521 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-179.ec2.internal\" not found"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.152641    2521 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-179.ec2.internal.18d62c217311c2a2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-179.ec2.internal.18d62c217311c2a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.ec2.internal,UID:ip-10-0-135-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-179.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.102437538 +0000 UTC m=+0.434287660,LastTimestamp:2026-09-17 17:33:44.103837892 +0000 UTC m=+0.435688014,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.ec2.internal,}"
Sep 17 17:33:44.249698 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.165050    2521 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-179.ec2.internal.18d62c217311eb7b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-179.ec2.internal.18d62c217311eb7b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.ec2.internal,UID:ip-10-0-135-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-179.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.102447995 +0000 UTC m=+0.434298119,LastTimestamp:2026-09-17 17:33:44.103847953 +0000 UTC m=+0.435698079,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.ec2.internal,}"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.179235    2521 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-179.ec2.internal.18d62c217597deee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-179.ec2.internal,UID:ip-10-0-135-179.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-135-179.ec2.internal,},FirstTimestamp:2026-09-17 17:33:44.144781038 +0000 UTC m=+0.476631151,LastTimestamp:2026-09-17 17:33:44.144781038 +0000 UTC m=+0.476631151,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-179.ec2.internal,}"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.180398    2521 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.181568    2521 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.181610    2521 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.181629    2521 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.181670    2521 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.194642    2521 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.243752    2521 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.244617    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.244650    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.244666    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:44.250241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.244691    2521 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.269768 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.269746    2521 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.269768 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.269769    2521 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-179.ec2.internal\": node \"ip-10-0-135-179.ec2.internal\" not found"
Sep 17 17:33:44.282681 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.282654    2521 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal"]
Sep 17 17:33:44.282803 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.282739    2521 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:44.283485 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.283471    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:44.283556 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.283497    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:44.283556 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.283507    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:44.285796 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.285784    2521 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:44.285915 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.285901    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.285963 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.285933    2521 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:44.286422 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.286408    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:44.286498 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.286431    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:44.286498 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.286411    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:44.286498 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.286444    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:44.286498 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.286464    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:44.286498 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.286481    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:44.288761 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.288745    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.288835 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.288776    2521 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:44.289433 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.289410    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:44.289524 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.289441    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:44.289524 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.289455    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:44.298903 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.298886    2521 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-179.ec2.internal\" not found"
Sep 17 17:33:44.316262 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.316246    2521 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-179.ec2.internal\" not found" node="ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.320818 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.320794    2521 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-179.ec2.internal\" not found" node="ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.399838 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.399808    2521 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-179.ec2.internal\" not found"
Sep 17 17:33:44.481053 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.481026    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c3b63325daf3988547716475abb1b98-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal\" (UID: \"5c3b63325daf3988547716475abb1b98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.481155 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.481056    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c3b63325daf3988547716475abb1b98-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal\" (UID: \"5c3b63325daf3988547716475abb1b98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.481155 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.481081    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75b092477877a2d07a79ef4fdc413311-config\") pod \"kube-apiserver-proxy-ip-10-0-135-179.ec2.internal\" (UID: \"75b092477877a2d07a79ef4fdc413311\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.500181 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.500125    2521 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-179.ec2.internal\" not found"
Sep 17 17:33:44.581875 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.581834    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75b092477877a2d07a79ef4fdc413311-config\") pod \"kube-apiserver-proxy-ip-10-0-135-179.ec2.internal\" (UID: \"75b092477877a2d07a79ef4fdc413311\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.581992 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.581880    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c3b63325daf3988547716475abb1b98-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal\" (UID: \"5c3b63325daf3988547716475abb1b98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.581992 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.581906    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c3b63325daf3988547716475abb1b98-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal\" (UID: \"5c3b63325daf3988547716475abb1b98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.581992 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.581948    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75b092477877a2d07a79ef4fdc413311-config\") pod \"kube-apiserver-proxy-ip-10-0-135-179.ec2.internal\" (UID: \"75b092477877a2d07a79ef4fdc413311\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.582106 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.581989    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c3b63325daf3988547716475abb1b98-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal\" (UID: \"5c3b63325daf3988547716475abb1b98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.582106 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.581957    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c3b63325daf3988547716475abb1b98-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal\" (UID: \"5c3b63325daf3988547716475abb1b98\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.600936 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.600914    2521 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-179.ec2.internal\" not found"
Sep 17 17:33:44.620088 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.620061    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.623888 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.623867    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:44.701190 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.701156    2521 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-179.ec2.internal\" not found"
Sep 17 17:33:44.801783 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:44.801682    2521 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-179.ec2.internal\" not found"
Sep 17 17:33:44.887555 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:44.887527    2521 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:46.266391 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.266367    2521 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:46.284569 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.284537    2521 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:33:46.285646 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.285630    2521 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:46.297675 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.297659    2521 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:33:46.297804 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.297783    2521 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:33:46.297861 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.297834    2521 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a38d827c26867415a91db529797da399-aa2f33587d936e11.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.135.179:46892->54.145.223.79:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal"
Sep 17 17:33:46.342984 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.342966    2521 apiserver.go:52] "Watching apiserver"
Sep 17 17:33:46.362404 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.362379    2521 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:33:46.364237 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.364211    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-nx4pl","kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rj29m","openshift-dns/node-resolver-89wfr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal","openshift-multus/multus-additional-cni-plugins-9zzms","openshift-multus/multus-tn7bn","kube-system/global-pull-secret-syncer-vzqnh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh","openshift-image-registry/node-ca-2rxr8","openshift-multus/network-metrics-daemon-n6dxl","openshift-network-diagnostics/network-check-target-h4vpg","openshift-network-operator/iptables-alerter-zpjxh","openshift-ovn-kubernetes/ovnkube-node-9qk8j"]
Sep 17 17:33:46.366452 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.366436    2521 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:33:46.366578 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.366553    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.368731 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.368712    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:46.368813 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.368790    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:33:46.368996 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.368979    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:33:46.369222 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.369204    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.369222 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.369217    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.369335 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.369278    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:33:46.369422 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.369406    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8ncd2\""
Sep 17 17:33:46.370767 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.370748    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-conf-dir\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.370818 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.370782    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-run-multus-certs\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.370818 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.370811    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-var-lib-kubelet\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.370897 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.370830    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.370995 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.370833    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-hostroot\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371039 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371020    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-daemon-config\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371090 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371058    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:46.371139 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371085    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmxm\" (UniqueName: \"kubernetes.io/projected/e8594596-9c6f-4e0f-9892-21e06eba0cad-kube-api-access-zdmxm\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:46.371139 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371117    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-cni-binary-copy\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371207 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371138    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-socket-dir-parent\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371207 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371152    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-var-lib-cni-bin\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371207 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371175    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-cni-dir\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371330 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371201    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-run-k8s-cni-cncf-io\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371330 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371235    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-run-netns\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371330 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371267    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-etc-kubernetes\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371330 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371303    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cpk\" (UniqueName: \"kubernetes.io/projected/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-kube-api-access-95cpk\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371336    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-system-cni-dir\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371362    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-cnibin\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.371484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.371389    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-var-lib-cni-multus\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.372915 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.372894    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9rm4q\""
Sep 17 17:33:46.373021 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.372897    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.373021 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.372939    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.373021 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.372948    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.375300 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.375280    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.375590 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.375533    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5bcg\""
Sep 17 17:33:46.375785 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.375769    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.375830 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.375769    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.377412 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.377397    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2t488\""
Sep 17 17:33:46.377472 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.377427    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:33:46.377472 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.377458    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.378357 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.378339    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.378802 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.378783    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:33:46.378802 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.378798    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:33:46.378941 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.378815    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:33:46.379251 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.379235    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.379964 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.379951    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.380156 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.380137    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:33:46.380379 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.380364    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:33:46.380379 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.380375    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-btd7d\""
Sep 17 17:33:46.382571 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.382539    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.382571 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.382552    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:33:46.382714 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.382576    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.382714 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.382601    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9gm2l\""
Sep 17 17:33:46.382972 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.382951    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.385415 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.385393    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:46.385600 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.385581    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d986h\""
Sep 17 17:33:46.385692 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.385628    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.385925 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.385905    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:33:46.385925 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.385916    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.386112 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.386067    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:33:46.387495 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.387461    2521 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:46.387718 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.387702    2521 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:33:46.387822 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.387801    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.390840 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.390820    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:33:46.390936 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.390922    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.391472 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.391458    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z9cx6\""
Sep 17 17:33:46.392434 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.391840    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.392434 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.392345    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:33:46.393118 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.393069    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:33:46.393444 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.393428    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:33:46.394304 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.394288    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:46.394392 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.394346    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:33:46.394540 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.394524    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hs7rn\""
Sep 17 17:33:46.432161 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.432145    2521 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9tww7"
Sep 17 17:33:46.445680 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.445660    2521 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9tww7"
Sep 17 17:33:46.467845 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.467824    2521 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:33:46.471635 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471616    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-node-log\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.471733 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471645    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-log-socket\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.471733 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471669    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-cni-netd\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.471733 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471721    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.471873 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471752    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-kubelet-config\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:46.471873 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471778    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-cnibin\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.471873 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471795    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-var-lib-cni-multus\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.471873 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471823    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-sys\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.471873 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471851    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-sys-fs\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.471873 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471862    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-cnibin\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.471873 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471854    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-var-lib-cni-multus\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472135 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471878    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3995ed8a-03ec-4c9a-b39f-014901ab9c47-hosts-file\") pod \"node-resolver-89wfr\" (UID: \"3995ed8a-03ec-4c9a-b39f-014901ab9c47\") " pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.472135 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471908    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmxm\" (UniqueName: \"kubernetes.io/projected/e8594596-9c6f-4e0f-9892-21e06eba0cad-kube-api-access-zdmxm\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:46.472135 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471931    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-systemd\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.472135 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471955    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-var-lib-kubelet\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.472135 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.471977    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-slash\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.472135 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472053    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-cni-bin\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.472135 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472090    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c529cb3-09aa-42d0-ae31-9b76df0ab375-cni-binary-copy\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.472135 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472131    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a18f3ab3-0d68-49b0-8189-1aa420dc674d-serviceca\") pod \"node-ca-2rxr8\" (UID: \"a18f3ab3-0d68-49b0-8189-1aa420dc674d\") " pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472150    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c431506-7b50-4923-931f-c07f448fa01e-agent-certs\") pod \"konnectivity-agent-nx4pl\" (UID: \"3c431506-7b50-4923-931f-c07f448fa01e\") " pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472174    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rzt2\" (UniqueName: \"kubernetes.io/projected/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-kube-api-access-6rzt2\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472191    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-env-overrides\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472205    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472220    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-registration-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472253    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-etc-kubernetes\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472268    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-tmp\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472283    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm2lb\" (UniqueName: \"kubernetes.io/projected/8c529cb3-09aa-42d0-ae31-9b76df0ab375-kube-api-access-qm2lb\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472300    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-etc-kubernetes\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472308    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszz7\" (UniqueName: \"kubernetes.io/projected/a80f5789-d1d3-4440-92e9-817b3f5d8f40-kube-api-access-wszz7\") pod \"iptables-alerter-zpjxh\" (UID: \"a80f5789-d1d3-4440-92e9-817b3f5d8f40\") " pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472330    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-conf-dir\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472365    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-conf-dir\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472384 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472367    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-run-multus-certs\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472400    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-socket-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472423    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c431506-7b50-4923-931f-c07f448fa01e-konnectivity-ca\") pod \"konnectivity-agent-nx4pl\" (UID: \"3c431506-7b50-4923-931f-c07f448fa01e\") " pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472456    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-modprobe-d\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472466    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-run-multus-certs\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472478    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472502    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472528    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3995ed8a-03ec-4c9a-b39f-014901ab9c47-tmp-dir\") pod \"node-resolver-89wfr\" (UID: \"3995ed8a-03ec-4c9a-b39f-014901ab9c47\") " pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472553    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-socket-dir-parent\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472600    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-run-netns\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472613    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-socket-dir-parent\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472624    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c529cb3-09aa-42d0-ae31-9b76df0ab375-system-cni-dir\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472659    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-run\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472678    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-run-netns\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472693    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-ovnkube-config\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472709    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-ovn-node-metrics-cert\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472746    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-run-netns\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.472788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472767    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-system-cni-dir\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472785    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-kubernetes\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472817    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-tuned\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472838    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c529cb3-09aa-42d0-ae31-9b76df0ab375-cnibin\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472859    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472882    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-system-cni-dir\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472898    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-var-lib-kubelet\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472931    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-hostroot\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472973    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-var-lib-kubelet\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472982    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-sysctl-d\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473001    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh876\" (UniqueName: \"kubernetes.io/projected/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-kube-api-access-dh876\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473016    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c529cb3-09aa-42d0-ae31-9b76df0ab375-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473030    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a80f5789-d1d3-4440-92e9-817b3f5d8f40-host-slash\") pod \"iptables-alerter-zpjxh\" (UID: \"a80f5789-d1d3-4440-92e9-817b3f5d8f40\") " pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.472982    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-hostroot\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473050    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a18f3ab3-0d68-49b0-8189-1aa420dc674d-host\") pod \"node-ca-2rxr8\" (UID: \"a18f3ab3-0d68-49b0-8189-1aa420dc674d\") " pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473074    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-var-lib-cni-bin\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473089    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c529cb3-09aa-42d0-ae31-9b76df0ab375-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.473484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473104    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c529cb3-09aa-42d0-ae31-9b76df0ab375-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473119    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473133    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-cni-dir\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473150    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-var-lib-cni-bin\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473156    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-run-k8s-cni-cncf-io\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473182    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-host-run-k8s-cni-cncf-io\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473192    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-lib-modules\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473198    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-cni-dir\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473225    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-systemd-units\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473243    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-etc-openvswitch\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473262    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-run-ovn\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473281    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-host\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473298    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-kubelet\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473311    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-var-lib-openvswitch\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473332    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psv67\" (UniqueName: \"kubernetes.io/projected/3995ed8a-03ec-4c9a-b39f-014901ab9c47-kube-api-access-psv67\") pod \"node-resolver-89wfr\" (UID: \"3995ed8a-03ec-4c9a-b39f-014901ab9c47\") " pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473361    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-daemon-config\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473389    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:46.474154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473412    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-run-openvswitch\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473435    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a80f5789-d1d3-4440-92e9-817b3f5d8f40-iptables-alerter-script\") pod \"iptables-alerter-zpjxh\" (UID: \"a80f5789-d1d3-4440-92e9-817b3f5d8f40\") " pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473458    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-cni-binary-copy\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473484    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-sysconfig\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473507    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-run-systemd\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473531    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-ovnkube-script-lib\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.473588    2521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.473654    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs podName:e8594596-9c6f-4e0f-9892-21e06eba0cad nodeName:}" failed. No retries permitted until 2026-09-17 17:33:46.973633952 +0000 UTC m=+2.018639039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs") pod "network-metrics-daemon-n6dxl" (UID: "e8594596-9c6f-4e0f-9892-21e06eba0cad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473593    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-device-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473691    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rjcl\" (UniqueName: \"kubernetes.io/projected/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-kube-api-access-7rjcl\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473716    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-dbus\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473740    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb8t5\" (UniqueName: \"kubernetes.io/projected/a18f3ab3-0d68-49b0-8189-1aa420dc674d-kube-api-access-jb8t5\") pod \"node-ca-2rxr8\" (UID: \"a18f3ab3-0d68-49b0-8189-1aa420dc674d\") " pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473781    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cpk\" (UniqueName: \"kubernetes.io/projected/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-kube-api-access-95cpk\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473909    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-multus-daemon-config\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473918    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-sysctl-conf\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.474758 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.473954    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-cni-binary-copy\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.480408 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.480390    2521 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:33:46.483886 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.483775    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmxm\" (UniqueName: \"kubernetes.io/projected/e8594596-9c6f-4e0f-9892-21e06eba0cad-kube-api-access-zdmxm\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:46.486260 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.486242    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cpk\" (UniqueName: \"kubernetes.io/projected/6b7acfa2-1f97-45c6-934a-d9b9b6098c9d-kube-api-access-95cpk\") pod \"multus-tn7bn\" (UID: \"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d\") " pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.499410 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.499376    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b092477877a2d07a79ef4fdc413311.slice/crio-9ccb592b1ad5cd8123a5e043de91e98caf348cc6a3d15cc1eff5bf1067185069 WatchSource:0}: Error finding container 9ccb592b1ad5cd8123a5e043de91e98caf348cc6a3d15cc1eff5bf1067185069: Status 404 returned error can't find the container with id 9ccb592b1ad5cd8123a5e043de91e98caf348cc6a3d15cc1eff5bf1067185069
Sep 17 17:33:46.500818 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.500798    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3b63325daf3988547716475abb1b98.slice/crio-7d536aede8dfbf4b5b8ffa36609adecd4dabe6f10c88d980a0150ac886d6defb WatchSource:0}: Error finding container 7d536aede8dfbf4b5b8ffa36609adecd4dabe6f10c88d980a0150ac886d6defb: Status 404 returned error can't find the container with id 7d536aede8dfbf4b5b8ffa36609adecd4dabe6f10c88d980a0150ac886d6defb
Sep 17 17:33:46.505860 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.505841    2521 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:33:46.539905 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.539867    2521 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:28:45 +0000 UTC" deadline="2028-02-12 23:27:45.678275012 +0000 UTC"
Sep 17 17:33:46.539905 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.539901    2521 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12317h53m59.138376035s"
Sep 17 17:33:46.574255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574232    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-var-lib-openvswitch\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.574366 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574260    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psv67\" (UniqueName: \"kubernetes.io/projected/3995ed8a-03ec-4c9a-b39f-014901ab9c47-kube-api-access-psv67\") pod \"node-resolver-89wfr\" (UID: \"3995ed8a-03ec-4c9a-b39f-014901ab9c47\") " pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.574366 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574285    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-run-openvswitch\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.574366 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574300    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a80f5789-d1d3-4440-92e9-817b3f5d8f40-iptables-alerter-script\") pod \"iptables-alerter-zpjxh\" (UID: \"a80f5789-d1d3-4440-92e9-817b3f5d8f40\") " pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.574366 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574315    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-sysconfig\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.574366 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574329    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-run-systemd\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.574366 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574341    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-var-lib-openvswitch\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.574366 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574368    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-run-systemd\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574376    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-run-openvswitch\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574395    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-sysconfig\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574452    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-ovnkube-script-lib\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574471    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-device-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574493    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rjcl\" (UniqueName: \"kubernetes.io/projected/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-kube-api-access-7rjcl\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574510    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-dbus\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574538    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb8t5\" (UniqueName: \"kubernetes.io/projected/a18f3ab3-0d68-49b0-8189-1aa420dc674d-kube-api-access-jb8t5\") pod \"node-ca-2rxr8\" (UID: \"a18f3ab3-0d68-49b0-8189-1aa420dc674d\") " pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574605    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-device-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574609    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-sysctl-conf\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.574641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574643    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-dbus\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574642    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-node-log\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574672    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-log-socket\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574679    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-node-log\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574719    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-log-socket\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574752    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-cni-netd\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574782    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574788    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-cni-netd\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574791    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-sysctl-conf\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574807    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-kubelet-config\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574840    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-sys\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574842    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574863    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-sys-fs\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574865    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-kubelet-config\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574892    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-sys\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574903    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3995ed8a-03ec-4c9a-b39f-014901ab9c47-hosts-file\") pod \"node-resolver-89wfr\" (UID: \"3995ed8a-03ec-4c9a-b39f-014901ab9c47\") " pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574916    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a80f5789-d1d3-4440-92e9-817b3f5d8f40-iptables-alerter-script\") pod \"iptables-alerter-zpjxh\" (UID: \"a80f5789-d1d3-4440-92e9-817b3f5d8f40\") " pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.575050 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574929    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-systemd\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574940    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3995ed8a-03ec-4c9a-b39f-014901ab9c47-hosts-file\") pod \"node-resolver-89wfr\" (UID: \"3995ed8a-03ec-4c9a-b39f-014901ab9c47\") " pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574923    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-sys-fs\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574954    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-var-lib-kubelet\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574972    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-slash\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.574986    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-cni-bin\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575011    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-systemd\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575022    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-ovnkube-script-lib\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575026    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c529cb3-09aa-42d0-ae31-9b76df0ab375-cni-binary-copy\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575037    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-var-lib-kubelet\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575061    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a18f3ab3-0d68-49b0-8189-1aa420dc674d-serviceca\") pod \"node-ca-2rxr8\" (UID: \"a18f3ab3-0d68-49b0-8189-1aa420dc674d\") " pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575077    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-cni-bin\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575086    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c431506-7b50-4923-931f-c07f448fa01e-agent-certs\") pod \"konnectivity-agent-nx4pl\" (UID: \"3c431506-7b50-4923-931f-c07f448fa01e\") " pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575111    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rzt2\" (UniqueName: \"kubernetes.io/projected/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-kube-api-access-6rzt2\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575250    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-env-overrides\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575049    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-slash\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575522    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a18f3ab3-0d68-49b0-8189-1aa420dc674d-serviceca\") pod \"node-ca-2rxr8\" (UID: \"a18f3ab3-0d68-49b0-8189-1aa420dc674d\") " pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.575742 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575660    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-env-overrides\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575870    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575900    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-registration-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575924    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-tmp\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575946    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm2lb\" (UniqueName: \"kubernetes.io/projected/8c529cb3-09aa-42d0-ae31-9b76df0ab375-kube-api-access-qm2lb\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575956    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575946    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c529cb3-09aa-42d0-ae31-9b76df0ab375-cni-binary-copy\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.575966    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszz7\" (UniqueName: \"kubernetes.io/projected/a80f5789-d1d3-4440-92e9-817b3f5d8f40-kube-api-access-wszz7\") pod \"iptables-alerter-zpjxh\" (UID: \"a80f5789-d1d3-4440-92e9-817b3f5d8f40\") " pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576001    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-socket-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576008    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-registration-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576029    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c431506-7b50-4923-931f-c07f448fa01e-konnectivity-ca\") pod \"konnectivity-agent-nx4pl\" (UID: \"3c431506-7b50-4923-931f-c07f448fa01e\") " pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576057    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-modprobe-d\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576101    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576124    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576146    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-socket-dir\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576149    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3995ed8a-03ec-4c9a-b39f-014901ab9c47-tmp-dir\") pod \"node-resolver-89wfr\" (UID: \"3995ed8a-03ec-4c9a-b39f-014901ab9c47\") " pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576191    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-modprobe-d\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.576406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576205    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-run-netns\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576232    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c529cb3-09aa-42d0-ae31-9b76df0ab375-system-cni-dir\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576243    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.576281    2521 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576300    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c529cb3-09aa-42d0-ae31-9b76df0ab375-system-cni-dir\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576299    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-run-netns\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576329    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-run\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.576339    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret podName:2da9dc7f-e3f7-442b-bf4e-badb16e819bd nodeName:}" failed. No retries permitted until 2026-09-17 17:33:47.07632216 +0000 UTC m=+2.121327234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret") pod "global-pull-secret-syncer-vzqnh" (UID: "2da9dc7f-e3f7-442b-bf4e-badb16e819bd") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576376    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-run\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576382    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-ovnkube-config\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576391    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3995ed8a-03ec-4c9a-b39f-014901ab9c47-tmp-dir\") pod \"node-resolver-89wfr\" (UID: \"3995ed8a-03ec-4c9a-b39f-014901ab9c47\") " pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576410    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-ovn-node-metrics-cert\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576444    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-kubernetes\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576507    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-kubernetes\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576685    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-tuned\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576717    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c529cb3-09aa-42d0-ae31-9b76df0ab375-cnibin\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576744    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.577216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576773    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-sysctl-d\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576789    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c529cb3-09aa-42d0-ae31-9b76df0ab375-cnibin\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576799    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh876\" (UniqueName: \"kubernetes.io/projected/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-kube-api-access-dh876\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576823    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c529cb3-09aa-42d0-ae31-9b76df0ab375-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576848    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a80f5789-d1d3-4440-92e9-817b3f5d8f40-host-slash\") pod \"iptables-alerter-zpjxh\" (UID: \"a80f5789-d1d3-4440-92e9-817b3f5d8f40\") " pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576870    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a18f3ab3-0d68-49b0-8189-1aa420dc674d-host\") pod \"node-ca-2rxr8\" (UID: \"a18f3ab3-0d68-49b0-8189-1aa420dc674d\") " pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576897    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c529cb3-09aa-42d0-ae31-9b76df0ab375-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576924    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c529cb3-09aa-42d0-ae31-9b76df0ab375-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576934    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-sysctl-d\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576949    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576976    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-lib-modules\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.576998    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a80f5789-d1d3-4440-92e9-817b3f5d8f40-host-slash\") pod \"iptables-alerter-zpjxh\" (UID: \"a80f5789-d1d3-4440-92e9-817b3f5d8f40\") " pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577009    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-ovnkube-config\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577001    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-systemd-units\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577042    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-systemd-units\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577048    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a18f3ab3-0d68-49b0-8189-1aa420dc674d-host\") pod \"node-ca-2rxr8\" (UID: \"a18f3ab3-0d68-49b0-8189-1aa420dc674d\") " pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577043    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c431506-7b50-4923-931f-c07f448fa01e-konnectivity-ca\") pod \"konnectivity-agent-nx4pl\" (UID: \"3c431506-7b50-4923-931f-c07f448fa01e\") " pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:33:46.577903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577091    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-etc-openvswitch\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577122    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-run-ovn\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577154    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c529cb3-09aa-42d0-ae31-9b76df0ab375-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577168    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-host\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577200    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-kubelet\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577263    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-host\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577203    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-etc-openvswitch\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577231    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-run-ovn\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577312    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-host-kubelet\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577465    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577592    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8c529cb3-09aa-42d0-ae31-9b76df0ab375-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577606    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8c529cb3-09aa-42d0-ae31-9b76df0ab375-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.578429 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.577593    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-lib-modules\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.578802 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.578538    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c431506-7b50-4923-931f-c07f448fa01e-agent-certs\") pod \"konnectivity-agent-nx4pl\" (UID: \"3c431506-7b50-4923-931f-c07f448fa01e\") " pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:33:46.578915 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.578896    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-tmp\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.579404 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.579389    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-etc-tuned\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.579480 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.579465    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-ovn-node-metrics-cert\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.585787 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.585531    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:46.585787 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.585541    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb8t5\" (UniqueName: \"kubernetes.io/projected/a18f3ab3-0d68-49b0-8189-1aa420dc674d-kube-api-access-jb8t5\") pod \"node-ca-2rxr8\" (UID: \"a18f3ab3-0d68-49b0-8189-1aa420dc674d\") " pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.585787 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.585583    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:46.585787 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.585596    2521 projected.go:196] Error preparing data for projected volume kube-api-access-qdrdr for pod openshift-network-diagnostics/network-check-target-h4vpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:46.585787 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.585655    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr podName:3e16cb48-397c-45e8-94c0-e40e16b52d8c nodeName:}" failed. No retries permitted until 2026-09-17 17:33:47.085641048 +0000 UTC m=+2.130646117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qdrdr" (UniqueName: "kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr") pod "network-check-target-h4vpg" (UID: "3e16cb48-397c-45e8-94c0-e40e16b52d8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:46.586169 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.586146    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rjcl\" (UniqueName: \"kubernetes.io/projected/4ee2a5ca-d56f-4930-96c3-d9c49f64015f-kube-api-access-7rjcl\") pod \"aws-ebs-csi-driver-node-lkskh\" (UID: \"4ee2a5ca-d56f-4930-96c3-d9c49f64015f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.586219 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.586149    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psv67\" (UniqueName: \"kubernetes.io/projected/3995ed8a-03ec-4c9a-b39f-014901ab9c47-kube-api-access-psv67\") pod \"node-resolver-89wfr\" (UID: \"3995ed8a-03ec-4c9a-b39f-014901ab9c47\") " pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.587459 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.587433    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszz7\" (UniqueName: \"kubernetes.io/projected/a80f5789-d1d3-4440-92e9-817b3f5d8f40-kube-api-access-wszz7\") pod \"iptables-alerter-zpjxh\" (UID: \"a80f5789-d1d3-4440-92e9-817b3f5d8f40\") " pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.587639 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.587623    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm2lb\" (UniqueName: \"kubernetes.io/projected/8c529cb3-09aa-42d0-ae31-9b76df0ab375-kube-api-access-qm2lb\") pod \"multus-additional-cni-plugins-9zzms\" (UID: \"8c529cb3-09aa-42d0-ae31-9b76df0ab375\") " pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.587933 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.587912    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rzt2\" (UniqueName: \"kubernetes.io/projected/5b48a73b-06fe-4f54-b05c-d37b3856cf6f-kube-api-access-6rzt2\") pod \"tuned-rj29m\" (UID: \"5b48a73b-06fe-4f54-b05c-d37b3856cf6f\") " pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.587975 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.587912    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh876\" (UniqueName: \"kubernetes.io/projected/c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6-kube-api-access-dh876\") pod \"ovnkube-node-9qk8j\" (UID: \"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.689956 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.689887    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tn7bn"
Sep 17 17:33:46.699732 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.699708    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7acfa2_1f97_45c6_934a_d9b9b6098c9d.slice/crio-15bb0f4b119b3895482c5018bb51799aaacf72785f0db5d52dbf1a30b31230d2 WatchSource:0}: Error finding container 15bb0f4b119b3895482c5018bb51799aaacf72785f0db5d52dbf1a30b31230d2: Status 404 returned error can't find the container with id 15bb0f4b119b3895482c5018bb51799aaacf72785f0db5d52dbf1a30b31230d2
Sep 17 17:33:46.705720 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.705702    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rj29m"
Sep 17 17:33:46.713162 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.713145    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b48a73b_06fe_4f54_b05c_d37b3856cf6f.slice/crio-88f185d39b81dbf747df99e9c0a4822f7afc99583a945301b706c94163ffc7e7 WatchSource:0}: Error finding container 88f185d39b81dbf747df99e9c0a4822f7afc99583a945301b706c94163ffc7e7: Status 404 returned error can't find the container with id 88f185d39b81dbf747df99e9c0a4822f7afc99583a945301b706c94163ffc7e7
Sep 17 17:33:46.720264 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.720243    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-89wfr"
Sep 17 17:33:46.722795 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.722780    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:33:46.728453 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.728434    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3995ed8a_03ec_4c9a_b39f_014901ab9c47.slice/crio-574b4bed919e65fe2ea2b792f01b1a5113d051ebaed496071be890b44c93822b WatchSource:0}: Error finding container 574b4bed919e65fe2ea2b792f01b1a5113d051ebaed496071be890b44c93822b: Status 404 returned error can't find the container with id 574b4bed919e65fe2ea2b792f01b1a5113d051ebaed496071be890b44c93822b
Sep 17 17:33:46.731040 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.731017    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a91a86_eab6_46ef_b4ec_713c7bdb1bd6.slice/crio-5d5e8e7c18a6ef89559ea7d028182f146b5a165e9c01d975c097b4ddd68c68b3 WatchSource:0}: Error finding container 5d5e8e7c18a6ef89559ea7d028182f146b5a165e9c01d975c097b4ddd68c68b3: Status 404 returned error can't find the container with id 5d5e8e7c18a6ef89559ea7d028182f146b5a165e9c01d975c097b4ddd68c68b3
Sep 17 17:33:46.738393 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.738373    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9zzms"
Sep 17 17:33:46.747337 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.747318    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c529cb3_09aa_42d0_ae31_9b76df0ab375.slice/crio-58ba3691fee4adae71fa47e41263b6d28a994ff6d4a0612101ba0739c163c6cd WatchSource:0}: Error finding container 58ba3691fee4adae71fa47e41263b6d28a994ff6d4a0612101ba0739c163c6cd: Status 404 returned error can't find the container with id 58ba3691fee4adae71fa47e41263b6d28a994ff6d4a0612101ba0739c163c6cd
Sep 17 17:33:46.754169 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.754153    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh"
Sep 17 17:33:46.761520 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.761503    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee2a5ca_d56f_4930_96c3_d9c49f64015f.slice/crio-59b75cb13cd163e4e343c48052c8fc699fed56f739dd32c28e69152d6bca611e WatchSource:0}: Error finding container 59b75cb13cd163e4e343c48052c8fc699fed56f739dd32c28e69152d6bca611e: Status 404 returned error can't find the container with id 59b75cb13cd163e4e343c48052c8fc699fed56f739dd32c28e69152d6bca611e
Sep 17 17:33:46.771915 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.771900    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpjxh"
Sep 17 17:33:46.779570 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.779545    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80f5789_d1d3_4440_92e9_817b3f5d8f40.slice/crio-b5249485c5f04645c7720b1c1f7f9488b5bea87befa002216632d4882f9df4b4 WatchSource:0}: Error finding container b5249485c5f04645c7720b1c1f7f9488b5bea87befa002216632d4882f9df4b4: Status 404 returned error can't find the container with id b5249485c5f04645c7720b1c1f7f9488b5bea87befa002216632d4882f9df4b4
Sep 17 17:33:46.785506 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.785490    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2rxr8"
Sep 17 17:33:46.794266 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.794248    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18f3ab3_0d68_49b0_8189_1aa420dc674d.slice/crio-6016004b3d252a601fdc001adbc28e10ef1742ef9cc0a3ff4f8caa8f8a01b5b9 WatchSource:0}: Error finding container 6016004b3d252a601fdc001adbc28e10ef1742ef9cc0a3ff4f8caa8f8a01b5b9: Status 404 returned error can't find the container with id 6016004b3d252a601fdc001adbc28e10ef1742ef9cc0a3ff4f8caa8f8a01b5b9
Sep 17 17:33:46.797026 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.797009    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:33:46.804588 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:33:46.804568    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c431506_7b50_4923_931f_c07f448fa01e.slice/crio-64e9b0e0db57dbde3ad96b861122ec60d5552c6342d927b68098afce44d887a0 WatchSource:0}: Error finding container 64e9b0e0db57dbde3ad96b861122ec60d5552c6342d927b68098afce44d887a0: Status 404 returned error can't find the container with id 64e9b0e0db57dbde3ad96b861122ec60d5552c6342d927b68098afce44d887a0
Sep 17 17:33:46.950688 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.950608    2521 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:46.980472 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:46.980440    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:46.980640 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.980589    2521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:46.980709 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:46.980659    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs podName:e8594596-9c6f-4e0f-9892-21e06eba0cad nodeName:}" failed. No retries permitted until 2026-09-17 17:33:47.980638332 +0000 UTC m=+3.025643410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs") pod "network-metrics-daemon-n6dxl" (UID: "e8594596-9c6f-4e0f-9892-21e06eba0cad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:47.049599 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.049569    2521 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:47.081117 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.081087    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:47.081256 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:47.081193    2521 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:47.081256 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:47.081241    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret podName:2da9dc7f-e3f7-442b-bf4e-badb16e819bd nodeName:}" failed. No retries permitted until 2026-09-17 17:33:48.081225179 +0000 UTC m=+3.126230259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret") pod "global-pull-secret-syncer-vzqnh" (UID: "2da9dc7f-e3f7-442b-bf4e-badb16e819bd") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:47.181908 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.181874    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:47.182062 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:47.182048    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:47.182130 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:47.182075    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:47.182130 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:47.182087    2521 projected.go:196] Error preparing data for projected volume kube-api-access-qdrdr for pod openshift-network-diagnostics/network-check-target-h4vpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:47.182231 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:47.182143    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr podName:3e16cb48-397c-45e8-94c0-e40e16b52d8c nodeName:}" failed. No retries permitted until 2026-09-17 17:33:48.182125691 +0000 UTC m=+3.227130763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdrdr" (UniqueName: "kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr") pod "network-check-target-h4vpg" (UID: "3e16cb48-397c-45e8-94c0-e40e16b52d8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:47.449096 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.449007    2521 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:28:46 +0000 UTC" deadline="2028-03-12 19:34:50.485976112 +0000 UTC"
Sep 17 17:33:47.449096 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.449051    2521 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13010h1m3.036930009s"
Sep 17 17:33:47.492628 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.492572    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" event={"ID":"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6","Type":"ContainerStarted","Data":"5d5e8e7c18a6ef89559ea7d028182f146b5a165e9c01d975c097b4ddd68c68b3"}
Sep 17 17:33:47.529717 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.529683    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-89wfr" event={"ID":"3995ed8a-03ec-4c9a-b39f-014901ab9c47","Type":"ContainerStarted","Data":"574b4bed919e65fe2ea2b792f01b1a5113d051ebaed496071be890b44c93822b"}
Sep 17 17:33:47.551703 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.551627    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rj29m" event={"ID":"5b48a73b-06fe-4f54-b05c-d37b3856cf6f","Type":"ContainerStarted","Data":"88f185d39b81dbf747df99e9c0a4822f7afc99583a945301b706c94163ffc7e7"}
Sep 17 17:33:47.570749 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.570714    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tn7bn" event={"ID":"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d","Type":"ContainerStarted","Data":"15bb0f4b119b3895482c5018bb51799aaacf72785f0db5d52dbf1a30b31230d2"}
Sep 17 17:33:47.578848 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.578818    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal" event={"ID":"5c3b63325daf3988547716475abb1b98","Type":"ContainerStarted","Data":"7d536aede8dfbf4b5b8ffa36609adecd4dabe6f10c88d980a0150ac886d6defb"}
Sep 17 17:33:47.584746 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.584719    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nx4pl" event={"ID":"3c431506-7b50-4923-931f-c07f448fa01e","Type":"ContainerStarted","Data":"64e9b0e0db57dbde3ad96b861122ec60d5552c6342d927b68098afce44d887a0"}
Sep 17 17:33:47.610895 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.610861    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpjxh" event={"ID":"a80f5789-d1d3-4440-92e9-817b3f5d8f40","Type":"ContainerStarted","Data":"b5249485c5f04645c7720b1c1f7f9488b5bea87befa002216632d4882f9df4b4"}
Sep 17 17:33:47.618386 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.618278    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9zzms" event={"ID":"8c529cb3-09aa-42d0-ae31-9b76df0ab375","Type":"ContainerStarted","Data":"58ba3691fee4adae71fa47e41263b6d28a994ff6d4a0612101ba0739c163c6cd"}
Sep 17 17:33:47.621361 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.621315    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal" event={"ID":"75b092477877a2d07a79ef4fdc413311","Type":"ContainerStarted","Data":"9ccb592b1ad5cd8123a5e043de91e98caf348cc6a3d15cc1eff5bf1067185069"}
Sep 17 17:33:47.625638 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.625570    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2rxr8" event={"ID":"a18f3ab3-0d68-49b0-8189-1aa420dc674d","Type":"ContainerStarted","Data":"6016004b3d252a601fdc001adbc28e10ef1742ef9cc0a3ff4f8caa8f8a01b5b9"}
Sep 17 17:33:47.640218 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.639952    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh" event={"ID":"4ee2a5ca-d56f-4930-96c3-d9c49f64015f","Type":"ContainerStarted","Data":"59b75cb13cd163e4e343c48052c8fc699fed56f739dd32c28e69152d6bca611e"}
Sep 17 17:33:47.988857 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:47.988786    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:47.989052 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:47.988997    2521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:47.989113 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:47.989054    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs podName:e8594596-9c6f-4e0f-9892-21e06eba0cad nodeName:}" failed. No retries permitted until 2026-09-17 17:33:49.989036685 +0000 UTC m=+5.034041757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs") pod "network-metrics-daemon-n6dxl" (UID: "e8594596-9c6f-4e0f-9892-21e06eba0cad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:48.089729 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:48.089683    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:48.089910 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:48.089865    2521 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:48.089974 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:48.089920    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret podName:2da9dc7f-e3f7-442b-bf4e-badb16e819bd nodeName:}" failed. No retries permitted until 2026-09-17 17:33:50.089903523 +0000 UTC m=+5.134908593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret") pod "global-pull-secret-syncer-vzqnh" (UID: "2da9dc7f-e3f7-442b-bf4e-badb16e819bd") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:48.191011 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:48.190979    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:48.191178 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:48.191146    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:48.191178 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:48.191164    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:48.191178 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:48.191176    2521 projected.go:196] Error preparing data for projected volume kube-api-access-qdrdr for pod openshift-network-diagnostics/network-check-target-h4vpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:48.191348 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:48.191231    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr podName:3e16cb48-397c-45e8-94c0-e40e16b52d8c nodeName:}" failed. No retries permitted until 2026-09-17 17:33:50.191213897 +0000 UTC m=+5.236218976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdrdr" (UniqueName: "kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr") pod "network-check-target-h4vpg" (UID: "3e16cb48-397c-45e8-94c0-e40e16b52d8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:48.449972 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:48.449863    2521 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:28:46 +0000 UTC" deadline="2028-03-21 17:26:51.029795836 +0000 UTC"
Sep 17 17:33:48.449972 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:48.449926    2521 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13223h53m2.579874515s"
Sep 17 17:33:48.469204 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:48.469177    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:48.469342 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:48.469290    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:48.469342 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:48.469318    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:33:48.469447 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:48.469424    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:33:48.469497 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:48.469482    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:48.469614 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:48.469568    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:33:49.054430 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:49.054294    2521 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:50.007412 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:50.007379    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:50.007884 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.007519    2521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:50.007884 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.007605    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs podName:e8594596-9c6f-4e0f-9892-21e06eba0cad nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.007585327 +0000 UTC m=+9.052590403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs") pod "network-metrics-daemon-n6dxl" (UID: "e8594596-9c6f-4e0f-9892-21e06eba0cad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:50.108576 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:50.108534    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:50.108790 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.108743    2521 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:50.108850 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.108799    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret podName:2da9dc7f-e3f7-442b-bf4e-badb16e819bd nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.108782522 +0000 UTC m=+9.153787590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret") pod "global-pull-secret-syncer-vzqnh" (UID: "2da9dc7f-e3f7-442b-bf4e-badb16e819bd") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:50.208967 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:50.208930    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:50.209192 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.209096    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:50.209192 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.209115    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:50.209192 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.209127    2521 projected.go:196] Error preparing data for projected volume kube-api-access-qdrdr for pod openshift-network-diagnostics/network-check-target-h4vpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:50.209192 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.209179    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr podName:3e16cb48-397c-45e8-94c0-e40e16b52d8c nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.209163596 +0000 UTC m=+9.254168678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdrdr" (UniqueName: "kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr") pod "network-check-target-h4vpg" (UID: "3e16cb48-397c-45e8-94c0-e40e16b52d8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:50.470235 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:50.469419    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:50.470235 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:50.469466    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:50.470235 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.469590    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:33:50.470235 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.470028    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:33:50.470235 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:50.470079    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:50.470235 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:50.470155    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:33:52.468857 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:52.468823    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:52.468857 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:52.468848    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:52.469479 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:52.468848    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:52.469479 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:52.468960    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:33:52.469479 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:52.469022    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:33:52.469479 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:52.469095    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:33:54.040378 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:54.040334    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:54.040924 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.040504    2521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:54.040924 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.040595    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs podName:e8594596-9c6f-4e0f-9892-21e06eba0cad nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.040573712 +0000 UTC m=+17.085578793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs") pod "network-metrics-daemon-n6dxl" (UID: "e8594596-9c6f-4e0f-9892-21e06eba0cad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:54.141410 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:54.141367    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:54.141592 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.141525    2521 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:54.141672 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.141612    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret podName:2da9dc7f-e3f7-442b-bf4e-badb16e819bd nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.141595708 +0000 UTC m=+17.186600779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret") pod "global-pull-secret-syncer-vzqnh" (UID: "2da9dc7f-e3f7-442b-bf4e-badb16e819bd") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:54.242474 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:54.242387    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:54.242649 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.242525    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:54.242649 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.242543    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:54.242649 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.242567    2521 projected.go:196] Error preparing data for projected volume kube-api-access-qdrdr for pod openshift-network-diagnostics/network-check-target-h4vpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:54.242649 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.242622    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr podName:3e16cb48-397c-45e8-94c0-e40e16b52d8c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.242604171 +0000 UTC m=+17.287609245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdrdr" (UniqueName: "kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr") pod "network-check-target-h4vpg" (UID: "3e16cb48-397c-45e8-94c0-e40e16b52d8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:54.469358 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:54.469315    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:54.469513 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:54.469320    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:54.469513 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:54.469334    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:54.469639 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.469514    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:33:54.469695 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.469676    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:33:54.469788 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:54.469770    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:33:56.468919 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:56.468883    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:56.469270 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:56.468886    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:56.469270 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:56.468991    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:33:56.469270 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:56.468904    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:56.469270 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:56.469044    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:33:56.469270 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:56.469136    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:33:58.469690 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:58.469613    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:33:58.469690 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:58.469632    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:33:58.469690 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:33:58.469632    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:33:58.470197 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:58.469731    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:33:58.470197 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:58.469800    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:33:58.470197 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:33:58.469869    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:00.468700 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:00.468638    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:00.469148 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:00.468762    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:00.469148 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:00.468771    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:00.469148 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:00.468802    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:00.469148 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:00.468911    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:00.469148 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:00.469000    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:02.103720 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:02.103680    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:02.104161 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.103833    2521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:02.104161 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.103896    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs podName:e8594596-9c6f-4e0f-9892-21e06eba0cad nodeName:}" failed. No retries permitted until 2026-09-17 17:34:18.103881078 +0000 UTC m=+33.148886144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs") pod "network-metrics-daemon-n6dxl" (UID: "e8594596-9c6f-4e0f-9892-21e06eba0cad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:02.204507 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:02.204472    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:02.204701 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.204613    2521 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:02.204701 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.204663    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret podName:2da9dc7f-e3f7-442b-bf4e-badb16e819bd nodeName:}" failed. No retries permitted until 2026-09-17 17:34:18.204648188 +0000 UTC m=+33.249653254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret") pod "global-pull-secret-syncer-vzqnh" (UID: "2da9dc7f-e3f7-442b-bf4e-badb16e819bd") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:02.305084 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:02.305048    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:02.305237 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.305198    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:02.305237 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.305223    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:02.305237 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.305237    2521 projected.go:196] Error preparing data for projected volume kube-api-access-qdrdr for pod openshift-network-diagnostics/network-check-target-h4vpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:02.305366 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.305298    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr podName:3e16cb48-397c-45e8-94c0-e40e16b52d8c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:18.305281543 +0000 UTC m=+33.350286634 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdrdr" (UniqueName: "kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr") pod "network-check-target-h4vpg" (UID: "3e16cb48-397c-45e8-94c0-e40e16b52d8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:02.469202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:02.469125    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:02.469202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:02.469153    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:02.469202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:02.469161    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:02.469532 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.469233    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:02.469532 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.469327    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:02.469532 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:02.469417    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:04.469412 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:04.469384    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:04.470035 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:04.469381    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:04.470035 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:04.469507    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:04.470035 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:04.469383    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:04.470035 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:04.469579    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:04.470035 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:04.469645    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:05.698162 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:05.697868    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rj29m" event={"ID":"5b48a73b-06fe-4f54-b05c-d37b3856cf6f","Type":"ContainerStarted","Data":"94db9de902eb5863b7de589d410aad1aa2ee0f205ccb63c07f1bf7be9fb7c8c5"}
Sep 17 17:34:05.699581 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:05.699543    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tn7bn" event={"ID":"6b7acfa2-1f97-45c6-934a-d9b9b6098c9d","Type":"ContainerStarted","Data":"33ac92516e61bbdde290212d9b85a6e3bac2e651d30de94b7269d4a68953589d"}
Sep 17 17:34:05.705755 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:05.705686    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal" event={"ID":"75b092477877a2d07a79ef4fdc413311","Type":"ContainerStarted","Data":"5449d8c871ffc3d806b2627e2fb3d3ec0754743f54b874c6c09d130bd823e18b"}
Sep 17 17:34:05.715017 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:05.714864    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rj29m" podStartSLOduration=2.004353671 podStartE2EDuration="20.714845927s" totalImagesPullingTime="18.710492256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:05.714267668 +0000 UTC m=+20.759272757" watchObservedRunningTime="2026-09-17 17:34:05.714845927 +0000 UTC m=+20.759851028"
Sep 17 17:34:05.718903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:05.718834    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" event={"ID":"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6","Type":"ContainerStarted","Data":"526decf8067247d69de21fab3c6277040f26fea26c44e99721565d3bcde6406d"}
Sep 17 17:34:05.728984 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:05.727922    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-179.ec2.internal" podStartSLOduration=19.727906235 podStartE2EDuration="19.727906235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:05.727180835 +0000 UTC m=+20.772185925" watchObservedRunningTime="2026-09-17 17:34:05.727906235 +0000 UTC m=+20.772911325"
Sep 17 17:34:05.743726 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:05.743677    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tn7bn" podStartSLOduration=1.856828369 podStartE2EDuration="20.743658399s" totalImagesPullingTime="18.88683003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:05.743162055 +0000 UTC m=+20.788167170" watchObservedRunningTime="2026-09-17 17:34:05.743658399 +0000 UTC m=+20.788663488"
Sep 17 17:34:06.469131 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.468872    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:06.469267 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.468888    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:06.469267 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:06.469186    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:06.469411 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:06.469282    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:06.469411 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.468903    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:06.469411 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:06.469390    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:06.722920 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.722841    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2rxr8" event={"ID":"a18f3ab3-0d68-49b0-8189-1aa420dc674d","Type":"ContainerStarted","Data":"9eb4f09eac5c0a8572797bfc142d8e014aa280ec4d8d013814adae6544e0ca90"}
Sep 17 17:34:06.724595 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.724571    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh" event={"ID":"4ee2a5ca-d56f-4930-96c3-d9c49f64015f","Type":"ContainerStarted","Data":"ecaad5baa59a799a7bf2209206f1c77ee86852e820aa9fc232432dbcd8dd2f6c"}
Sep 17 17:34:06.727996 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.727973    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" event={"ID":"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6","Type":"ContainerStarted","Data":"36d6bcb2676e15fb7d551eafe33cc2f05ad307d1f7ca7ccad9649a837df427ed"}
Sep 17 17:34:06.728090 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.728047    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" event={"ID":"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6","Type":"ContainerStarted","Data":"c6f77a14852883f5668363eb3bcaf6eebd77d83c6125072dfd968a1736941267"}
Sep 17 17:34:06.728090 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.728083    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" event={"ID":"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6","Type":"ContainerStarted","Data":"e42e7a61e5ae3e6251f78fca6003127a8bdf3075ef54c844e0dd360f74f39486"}
Sep 17 17:34:06.728195 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.728128    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" event={"ID":"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6","Type":"ContainerStarted","Data":"47ac6e2d10d0628a757abccc921a8af5714a1838474bf81505b830b6a89f2e1d"}
Sep 17 17:34:06.728195 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.728157    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" event={"ID":"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6","Type":"ContainerStarted","Data":"6b025aa209dcd44309c1c568c2efa28db6397c0add41d5d769d5aa23aa7c551c"}
Sep 17 17:34:06.729499 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.729475    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-89wfr" event={"ID":"3995ed8a-03ec-4c9a-b39f-014901ab9c47","Type":"ContainerStarted","Data":"a3f79c3eabcd6038c6fa88e929f046609de2042f2c658501f6e2573d50737a55"}
Sep 17 17:34:06.731093 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.731068    2521 generic.go:356] "Generic (PLEG): container finished" podID="5c3b63325daf3988547716475abb1b98" containerID="fcb868222e447be969b7b5f4a3e73b265152b02166b9312dfbb0663a6f3f720e" exitCode=0
Sep 17 17:34:06.731182 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.731116    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal" event={"ID":"5c3b63325daf3988547716475abb1b98","Type":"ContainerDied","Data":"fcb868222e447be969b7b5f4a3e73b265152b02166b9312dfbb0663a6f3f720e"}
Sep 17 17:34:06.732835 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.732812    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nx4pl" event={"ID":"3c431506-7b50-4923-931f-c07f448fa01e","Type":"ContainerStarted","Data":"e05448e9ef2f04def2991a66efb9d95aa66a42a0f30f81f08c1cdcd0140d8a04"}
Sep 17 17:34:06.734425 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.734404    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpjxh" event={"ID":"a80f5789-d1d3-4440-92e9-817b3f5d8f40","Type":"ContainerStarted","Data":"57db7ba362079e8726fba5416087bf5270fd67d7cd763d8310ef8dd862f4ecf3"}
Sep 17 17:34:06.736005 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.735984    2521 generic.go:356] "Generic (PLEG): container finished" podID="8c529cb3-09aa-42d0-ae31-9b76df0ab375" containerID="ac7d211f03b8563ff8d64eb5e3fab55439a2350de157587f94c50b075af147d0" exitCode=0
Sep 17 17:34:06.736088 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.736010    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9zzms" event={"ID":"8c529cb3-09aa-42d0-ae31-9b76df0ab375","Type":"ContainerDied","Data":"ac7d211f03b8563ff8d64eb5e3fab55439a2350de157587f94c50b075af147d0"}
Sep 17 17:34:06.739290 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.739255    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2rxr8" podStartSLOduration=3.111559348 podStartE2EDuration="21.739239942s" totalImagesPullingTime="18.627680594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.738636771 +0000 UTC m=+21.783641861" watchObservedRunningTime="2026-09-17 17:34:06.739239942 +0000 UTC m=+21.784245022"
Sep 17 17:34:06.755811 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.755774    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zpjxh" podStartSLOduration=3.11176508 podStartE2EDuration="21.755761939s" totalImagesPullingTime="18.643996859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.755469093 +0000 UTC m=+21.800474181" watchObservedRunningTime="2026-09-17 17:34:06.755761939 +0000 UTC m=+21.800767029"
Sep 17 17:34:06.770500 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.770450    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-89wfr" podStartSLOduration=3.076854797 podStartE2EDuration="21.770438568s" totalImagesPullingTime="18.693583771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.769970229 +0000 UTC m=+21.814975319" watchObservedRunningTime="2026-09-17 17:34:06.770438568 +0000 UTC m=+21.815443658"
Sep 17 17:34:06.806104 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.806062    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nx4pl" podStartSLOduration=3.188677843 podStartE2EDuration="21.806045176s" totalImagesPullingTime="18.617367333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.80558271 +0000 UTC m=+21.850587799" watchObservedRunningTime="2026-09-17 17:34:06.806045176 +0000 UTC m=+21.851050265"
Sep 17 17:34:06.960379 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:06.960356    2521 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:07.375544 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.375514    2521 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:34:07.376111 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.376092    2521 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:34:07.447132 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.447027    2521 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:06.960376458Z","UUID":"66456fb7-6b19-461b-85ac-227d6c6a2fb4","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:07.449197 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.449176    2521 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:07.449331 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.449210    2521 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:07.740852 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.740778    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh" event={"ID":"4ee2a5ca-d56f-4930-96c3-d9c49f64015f","Type":"ContainerStarted","Data":"5c9ec8d5782de01a737305b1a463d854987b1acd89b2f80aa20d739eb2c8c63b"}
Sep 17 17:34:07.743030 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.743003    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal" event={"ID":"5c3b63325daf3988547716475abb1b98","Type":"ContainerStarted","Data":"a2d76c2f4e0980dd9f656a334ae6c41af1fa39fc68aac6aa78e2bde9a40fbaa1"}
Sep 17 17:34:07.743407 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.743376    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:34:07.744085 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.744067    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nx4pl"
Sep 17 17:34:07.762397 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:07.762349    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-179.ec2.internal" podStartSLOduration=21.762331433 podStartE2EDuration="21.762331433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:07.762137714 +0000 UTC m=+22.807142804" watchObservedRunningTime="2026-09-17 17:34:07.762331433 +0000 UTC m=+22.807336525"
Sep 17 17:34:08.468770 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:08.468698    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:08.468922 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:08.468801    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:08.468922 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:08.468805    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:08.468922 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:08.468824    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:08.469039 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:08.468920    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:08.469039 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:08.468991    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:08.747273 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:08.747199    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh" event={"ID":"4ee2a5ca-d56f-4930-96c3-d9c49f64015f","Type":"ContainerStarted","Data":"dd97e13a24a8fb9dc31a522dee2a469272840cebe0feb58a96c79ca941e22a58"}
Sep 17 17:34:08.750535 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:08.750510    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" event={"ID":"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6","Type":"ContainerStarted","Data":"1167fc31605da2b974d7f00ba933466709d2881098047e93e7a330cfb46b4be3"}
Sep 17 17:34:08.778108 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:08.778066    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lkskh" podStartSLOduration=10.924219551 podStartE2EDuration="23.778050991s" totalImagesPullingTime="12.85383144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:08.777583147 +0000 UTC m=+23.822588237" watchObservedRunningTime="2026-09-17 17:34:08.778050991 +0000 UTC m=+23.823056083"
Sep 17 17:34:10.468825 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:10.468792    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:10.468825 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:10.468818    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:10.468825 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:10.468827    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:10.469302 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:10.468906    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:10.469302 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:10.468970    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:10.469302 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:10.469052    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:11.758957 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:11.758793    2521 generic.go:356] "Generic (PLEG): container finished" podID="8c529cb3-09aa-42d0-ae31-9b76df0ab375" containerID="4f453feb3a10298a5b662ab51a272d234916a0f9872d103689fc4446cbd84c67" exitCode=0
Sep 17 17:34:11.759342 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:11.758878    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9zzms" event={"ID":"8c529cb3-09aa-42d0-ae31-9b76df0ab375","Type":"ContainerDied","Data":"4f453feb3a10298a5b662ab51a272d234916a0f9872d103689fc4446cbd84c67"}
Sep 17 17:34:11.762704 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:11.762682    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" event={"ID":"c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6","Type":"ContainerStarted","Data":"35ebec6466a5fd5925dcecd082c7be201da9b5564969a22f0f3afe00c5e49c05"}
Sep 17 17:34:11.762985 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:11.762961    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:34:11.777349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:11.777331    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:34:11.831638 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:11.831602    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j" podStartSLOduration=8.12922017 podStartE2EDuration="26.831590848s" totalImagesPullingTime="18.702370678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:11.830447765 +0000 UTC m=+26.875452855" watchObservedRunningTime="2026-09-17 17:34:11.831590848 +0000 UTC m=+26.876595937"
Sep 17 17:34:12.468954 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:12.468923    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:12.469096 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:12.468930    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:12.469096 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:12.469041    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:12.469195 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:12.468931    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:12.469195 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:12.469105    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:12.469195 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:12.469183    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:12.767785 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:12.767758    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9zzms" event={"ID":"8c529cb3-09aa-42d0-ae31-9b76df0ab375","Type":"ContainerStarted","Data":"57d2bf8781c246a86ebd27a9bea3850f5519f2e938de5bdf892a9457a999c07e"}
Sep 17 17:34:12.768378 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:12.768274    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:34:12.768378 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:12.768332    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:34:12.781897 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:12.781879    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:34:13.063387 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:13.063360    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4vpg"]
Sep 17 17:34:13.063487 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:13.063438    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:13.063532 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:13.063518    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:13.066326 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:13.066306    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n6dxl"]
Sep 17 17:34:13.066407 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:13.066377    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:13.066462 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:13.066447    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:13.068865 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:13.068674    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vzqnh"]
Sep 17 17:34:13.068865 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:13.068753    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:13.068865 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:13.068837    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:13.771550 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:13.771519    2521 generic.go:356] "Generic (PLEG): container finished" podID="8c529cb3-09aa-42d0-ae31-9b76df0ab375" containerID="57d2bf8781c246a86ebd27a9bea3850f5519f2e938de5bdf892a9457a999c07e" exitCode=0
Sep 17 17:34:13.772290 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:13.771591    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9zzms" event={"ID":"8c529cb3-09aa-42d0-ae31-9b76df0ab375","Type":"ContainerDied","Data":"57d2bf8781c246a86ebd27a9bea3850f5519f2e938de5bdf892a9457a999c07e"}
Sep 17 17:34:14.469493 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:14.469327    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:14.469604 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:14.469367    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:14.469604 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:14.469380    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:14.469678 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:14.469646    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:14.469745 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:14.469719    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:14.469816 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:14.469797    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:14.775732 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:14.775699    2521 generic.go:356] "Generic (PLEG): container finished" podID="8c529cb3-09aa-42d0-ae31-9b76df0ab375" containerID="b744fcbb27de42bad71cd0cfbef0fc406a46aae7497c961807c724f6a9b1676a" exitCode=0
Sep 17 17:34:14.776134 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:14.775790    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9zzms" event={"ID":"8c529cb3-09aa-42d0-ae31-9b76df0ab375","Type":"ContainerDied","Data":"b744fcbb27de42bad71cd0cfbef0fc406a46aae7497c961807c724f6a9b1676a"}
Sep 17 17:34:16.469081 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:16.469041    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:16.469514 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:16.469041    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:16.469514 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:16.469181    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:34:16.469514 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:16.469058    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:16.469514 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:16.469301    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4vpg" podUID="3e16cb48-397c-45e8-94c0-e40e16b52d8c"
Sep 17 17:34:16.469514 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:16.469310    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vzqnh" podUID="2da9dc7f-e3f7-442b-bf4e-badb16e819bd"
Sep 17 17:34:18.117047 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.117004    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:18.117486 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.117193    2521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:18.117486 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.117281    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs podName:e8594596-9c6f-4e0f-9892-21e06eba0cad nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.117258445 +0000 UTC m=+65.162263513 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs") pod "network-metrics-daemon-n6dxl" (UID: "e8594596-9c6f-4e0f-9892-21e06eba0cad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:18.218167 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.218122    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:18.218359 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.218270    2521 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:18.218359 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.218356    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret podName:2da9dc7f-e3f7-442b-bf4e-badb16e819bd nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.218333814 +0000 UTC m=+65.263338897 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret") pod "global-pull-secret-syncer-vzqnh" (UID: "2da9dc7f-e3f7-442b-bf4e-badb16e819bd") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:18.230842 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.230817    2521 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-179.ec2.internal" event="NodeReady"
Sep 17 17:34:18.231006 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.230969    2521 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:34:18.284703 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.284612    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"]
Sep 17 17:34:18.311674 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.311643    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bb59946d7-hbp89"]
Sep 17 17:34:18.311842 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.311777    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"
Sep 17 17:34:18.315213 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.315188    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:34:18.315373 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.315297    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:34:18.315765 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.315744    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:34:18.316118 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.316096    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-c6gn8\""
Sep 17 17:34:18.316215 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.316165    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 17:34:18.318467 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.318444    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:18.318615 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.318590    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:18.318678 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.318619    2521 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:18.318678 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.318629    2521 projected.go:196] Error preparing data for projected volume kube-api-access-qdrdr for pod openshift-network-diagnostics/network-check-target-h4vpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:18.318766 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.318703    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr podName:3e16cb48-397c-45e8-94c0-e40e16b52d8c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.318684179 +0000 UTC m=+65.363689259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qdrdr" (UniqueName: "kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr") pod "network-check-target-h4vpg" (UID: "3e16cb48-397c-45e8-94c0-e40e16b52d8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:18.331662 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.331637    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"]
Sep 17 17:34:18.331819 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.331801    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.336313 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.336286    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-trqlz\""
Sep 17 17:34:18.337020 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.336998    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:34:18.337163 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.337090    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:34:18.337327 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.337305    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:34:18.341235 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.341210    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:34:18.355689 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.355660    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"]
Sep 17 17:34:18.355853 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.355788    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.359500 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.358890    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 17:34:18.371874 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.371848    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"]
Sep 17 17:34:18.372200 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.371883    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6zpfd"]
Sep 17 17:34:18.372200 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.372000    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.374609 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.374581    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 17:34:18.374949 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.374930    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 17:34:18.375222 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.375203    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 17:34:18.375344 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.375324    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 17:34:18.390309 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.390274    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9hzc9"]
Sep 17 17:34:18.390469 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.390406    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.395152 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.395131    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:34:18.395426 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.395408    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:34:18.395551 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.395536    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdp8v\""
Sep 17 17:34:18.406123 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.406087    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"]
Sep 17 17:34:18.406258 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.406144    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bb59946d7-hbp89"]
Sep 17 17:34:18.406258 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.406166    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"]
Sep 17 17:34:18.406258 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.406180    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6zpfd"]
Sep 17 17:34:18.406258 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.406231    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9hzc9"
Sep 17 17:34:18.406426 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.406280    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9hzc9"]
Sep 17 17:34:18.409695 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.409660    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:34:18.409848 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.409825    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:34:18.409903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.409889    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8mqcm\""
Sep 17 17:34:18.410229 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.410210    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:34:18.418866 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.418840    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5252e4c9-2542-43d0-9262-fd67eb5eea6d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6548c74dcb-ntxjf\" (UID: \"5252e4c9-2542-43d0-9262-fd67eb5eea6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"
Sep 17 17:34:18.418975 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.418940    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-certificates\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.418975 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.418969    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-bound-sa-token\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.419084 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.419001    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.419084 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.419025    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-trusted-ca\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.419084 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.419044    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlj59\" (UniqueName: \"kubernetes.io/projected/5252e4c9-2542-43d0-9262-fd67eb5eea6d-kube-api-access-rlj59\") pod \"managed-serviceaccount-addon-agent-6548c74dcb-ntxjf\" (UID: \"5252e4c9-2542-43d0-9262-fd67eb5eea6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"
Sep 17 17:34:18.419245 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.419091    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-ca-trust-extracted\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.419245 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.419118    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-installation-pull-secrets\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.419245 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.419158    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-image-registry-private-configuration\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.419245 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.419191    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdm7w\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-kube-api-access-pdm7w\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.469395 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.469360    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:18.469614 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.469403    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:18.469614 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.469468    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:18.472202 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.472179    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:18.472569 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.472532    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-745tx\""
Sep 17 17:34:18.472813 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.472770    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7zxgv\""
Sep 17 17:34:18.472813 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.472804    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:18.472925 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.472859    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:18.473100 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.473085    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:18.519800 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.519767    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-image-registry-private-configuration\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.519942 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.519810    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdm7w\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-kube-api-access-pdm7w\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.519942 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.519840    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp4r6\" (UniqueName: \"kubernetes.io/projected/daad7b8f-2beb-46aa-91d7-cafffb2e11e4-kube-api-access-jp4r6\") pod \"ingress-canary-9hzc9\" (UID: \"daad7b8f-2beb-46aa-91d7-cafffb2e11e4\") " pod="openshift-ingress-canary/ingress-canary-9hzc9"
Sep 17 17:34:18.519942 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.519868    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-ca\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.519942 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.519925    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5252e4c9-2542-43d0-9262-fd67eb5eea6d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6548c74dcb-ntxjf\" (UID: \"5252e4c9-2542-43d0-9262-fd67eb5eea6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"
Sep 17 17:34:18.520152 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.519948    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/058158ca-fba0-4467-9c85-4661b64588ee-config-volume\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.520152 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.519994    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kkmx\" (UniqueName: \"kubernetes.io/projected/c3a67c51-fa23-48cc-b243-0d6ff5843cf9-kube-api-access-4kkmx\") pod \"klusterlet-addon-workmgr-67f48bcb4-vwv5r\" (UID: \"c3a67c51-fa23-48cc-b243-0d6ff5843cf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.520152 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520015    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-hub\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.520152 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520037    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/daad7b8f-2beb-46aa-91d7-cafffb2e11e4-cert\") pod \"ingress-canary-9hzc9\" (UID: \"daad7b8f-2beb-46aa-91d7-cafffb2e11e4\") " pod="openshift-ingress-canary/ingress-canary-9hzc9"
Sep 17 17:34:18.520152 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520109    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3a67c51-fa23-48cc-b243-0d6ff5843cf9-tmp\") pod \"klusterlet-addon-workmgr-67f48bcb4-vwv5r\" (UID: \"c3a67c51-fa23-48cc-b243-0d6ff5843cf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.520152 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520136    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c3a67c51-fa23-48cc-b243-0d6ff5843cf9-klusterlet-config\") pod \"klusterlet-addon-workmgr-67f48bcb4-vwv5r\" (UID: \"c3a67c51-fa23-48cc-b243-0d6ff5843cf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.520406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520171    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dggh8\" (UniqueName: \"kubernetes.io/projected/c2de2127-a021-4e72-b532-29117efb2909-kube-api-access-dggh8\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.520406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520214    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.520406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520255    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-certificates\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.520406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520279    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-bound-sa-token\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.520406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520308    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.520406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520327    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-trusted-ca\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.520406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520348    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlj59\" (UniqueName: \"kubernetes.io/projected/5252e4c9-2542-43d0-9262-fd67eb5eea6d-kube-api-access-rlj59\") pod \"managed-serviceaccount-addon-agent-6548c74dcb-ntxjf\" (UID: \"5252e4c9-2542-43d0-9262-fd67eb5eea6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"
Sep 17 17:34:18.520406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520370    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxmk\" (UniqueName: \"kubernetes.io/projected/058158ca-fba0-4467-9c85-4661b64588ee-kube-api-access-nqxmk\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.520406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520396    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-ca-trust-extracted\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.520757 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520418    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.520757 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520444    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-installation-pull-secrets\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.520757 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520465    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058158ca-fba0-4467-9c85-4661b64588ee-tmp-dir\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.520757 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520487    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2de2127-a021-4e72-b532-29117efb2909-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.520757 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520514    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.520924 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520836    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-certificates\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.520924 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.520904    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-ca-trust-extracted\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.520993 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.520925    2521 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:18.520993 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.520941    2521 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb59946d7-hbp89: secret "image-registry-tls" not found
Sep 17 17:34:18.521073 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.521002    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls podName:1b539bd7-2cb1-4894-acd7-0579cf3b5bc1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:19.020977192 +0000 UTC m=+34.065982265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls") pod "image-registry-6bb59946d7-hbp89" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1") : secret "image-registry-tls" not found
Sep 17 17:34:18.521399 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.521360    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-trusted-ca\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.524707 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.524637    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-image-registry-private-configuration\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.524707 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.524637    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-installation-pull-secrets\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.524847 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.524785    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5252e4c9-2542-43d0-9262-fd67eb5eea6d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6548c74dcb-ntxjf\" (UID: \"5252e4c9-2542-43d0-9262-fd67eb5eea6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"
Sep 17 17:34:18.532541 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.532469    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlj59\" (UniqueName: \"kubernetes.io/projected/5252e4c9-2542-43d0-9262-fd67eb5eea6d-kube-api-access-rlj59\") pod \"managed-serviceaccount-addon-agent-6548c74dcb-ntxjf\" (UID: \"5252e4c9-2542-43d0-9262-fd67eb5eea6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"
Sep 17 17:34:18.532667 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.532651    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdm7w\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-kube-api-access-pdm7w\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.534415 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.534396    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-bound-sa-token\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:18.621717 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.621632    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/058158ca-fba0-4467-9c85-4661b64588ee-config-volume\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.621717 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.621687    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kkmx\" (UniqueName: \"kubernetes.io/projected/c3a67c51-fa23-48cc-b243-0d6ff5843cf9-kube-api-access-4kkmx\") pod \"klusterlet-addon-workmgr-67f48bcb4-vwv5r\" (UID: \"c3a67c51-fa23-48cc-b243-0d6ff5843cf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.621937 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.621813    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-hub\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.621937 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.621854    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/daad7b8f-2beb-46aa-91d7-cafffb2e11e4-cert\") pod \"ingress-canary-9hzc9\" (UID: \"daad7b8f-2beb-46aa-91d7-cafffb2e11e4\") " pod="openshift-ingress-canary/ingress-canary-9hzc9"
Sep 17 17:34:18.621937 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.621892    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3a67c51-fa23-48cc-b243-0d6ff5843cf9-tmp\") pod \"klusterlet-addon-workmgr-67f48bcb4-vwv5r\" (UID: \"c3a67c51-fa23-48cc-b243-0d6ff5843cf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.621937 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.621909    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c3a67c51-fa23-48cc-b243-0d6ff5843cf9-klusterlet-config\") pod \"klusterlet-addon-workmgr-67f48bcb4-vwv5r\" (UID: \"c3a67c51-fa23-48cc-b243-0d6ff5843cf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.621937 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.621936    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dggh8\" (UniqueName: \"kubernetes.io/projected/c2de2127-a021-4e72-b532-29117efb2909-kube-api-access-dggh8\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.622228 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.621974    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.622228 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.622077    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxmk\" (UniqueName: \"kubernetes.io/projected/058158ca-fba0-4467-9c85-4661b64588ee-kube-api-access-nqxmk\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.622228 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.622109    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.622228 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.622137    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058158ca-fba0-4467-9c85-4661b64588ee-tmp-dir\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.622228 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.622163    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2de2127-a021-4e72-b532-29117efb2909-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.622228 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.622188    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.622534 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.622243    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp4r6\" (UniqueName: \"kubernetes.io/projected/daad7b8f-2beb-46aa-91d7-cafffb2e11e4-kube-api-access-jp4r6\") pod \"ingress-canary-9hzc9\" (UID: \"daad7b8f-2beb-46aa-91d7-cafffb2e11e4\") " pod="openshift-ingress-canary/ingress-canary-9hzc9"
Sep 17 17:34:18.622534 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.622272    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-ca\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.622534 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.622284    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3a67c51-fa23-48cc-b243-0d6ff5843cf9-tmp\") pod \"klusterlet-addon-workmgr-67f48bcb4-vwv5r\" (UID: \"c3a67c51-fa23-48cc-b243-0d6ff5843cf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.622534 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.622459    2521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:18.622534 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:18.622530    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls podName:058158ca-fba0-4467-9c85-4661b64588ee nodeName:}" failed. No retries permitted until 2026-09-17 17:34:19.122510372 +0000 UTC m=+34.167515459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls") pod "dns-default-6zpfd" (UID: "058158ca-fba0-4467-9c85-4661b64588ee") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:18.622931 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.622903    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/058158ca-fba0-4467-9c85-4661b64588ee-config-volume\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.623374 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.623291    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058158ca-fba0-4467-9c85-4661b64588ee-tmp-dir\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.623612 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.623587    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2de2127-a021-4e72-b532-29117efb2909-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.624933 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.624913    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/daad7b8f-2beb-46aa-91d7-cafffb2e11e4-cert\") pod \"ingress-canary-9hzc9\" (UID: \"daad7b8f-2beb-46aa-91d7-cafffb2e11e4\") " pod="openshift-ingress-canary/ingress-canary-9hzc9"
Sep 17 17:34:18.625274 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.625230    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-ca\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.625588 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.625547    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-hub\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.625997 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.625974    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.626249 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.626209    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c3a67c51-fa23-48cc-b243-0d6ff5843cf9-klusterlet-config\") pod \"klusterlet-addon-workmgr-67f48bcb4-vwv5r\" (UID: \"c3a67c51-fa23-48cc-b243-0d6ff5843cf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.626421 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.626403    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c2de2127-a021-4e72-b532-29117efb2909-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.630898 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.630880    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"
Sep 17 17:34:18.634520 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.634475    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxmk\" (UniqueName: \"kubernetes.io/projected/058158ca-fba0-4467-9c85-4661b64588ee-kube-api-access-nqxmk\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:18.635003 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.634964    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kkmx\" (UniqueName: \"kubernetes.io/projected/c3a67c51-fa23-48cc-b243-0d6ff5843cf9-kube-api-access-4kkmx\") pod \"klusterlet-addon-workmgr-67f48bcb4-vwv5r\" (UID: \"c3a67c51-fa23-48cc-b243-0d6ff5843cf9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.635167 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.635144    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp4r6\" (UniqueName: \"kubernetes.io/projected/daad7b8f-2beb-46aa-91d7-cafffb2e11e4-kube-api-access-jp4r6\") pod \"ingress-canary-9hzc9\" (UID: \"daad7b8f-2beb-46aa-91d7-cafffb2e11e4\") " pod="openshift-ingress-canary/ingress-canary-9hzc9"
Sep 17 17:34:18.635242 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.635178    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dggh8\" (UniqueName: \"kubernetes.io/projected/c2de2127-a021-4e72-b532-29117efb2909-kube-api-access-dggh8\") pod \"cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp\" (UID: \"c2de2127-a021-4e72-b532-29117efb2909\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.668884 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.668864    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:18.682030 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.682001    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:34:18.721991 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:18.721968    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9hzc9"
Sep 17 17:34:19.025249 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.024987    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:19.025334 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:19.025143    2521 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:19.025334 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:19.025307    2521 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb59946d7-hbp89: secret "image-registry-tls" not found
Sep 17 17:34:19.025431 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:19.025376    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls podName:1b539bd7-2cb1-4894-acd7-0579cf3b5bc1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.025360958 +0000 UTC m=+35.070366025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls") pod "image-registry-6bb59946d7-hbp89" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1") : secret "image-registry-tls" not found
Sep 17 17:34:19.027057 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.027033    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"]
Sep 17 17:34:19.028577 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.028542    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf"]
Sep 17 17:34:19.029631 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.029484    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"]
Sep 17 17:34:19.031731 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:34:19.031707    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5252e4c9_2542_43d0_9262_fd67eb5eea6d.slice/crio-9abf7babc3a8bc84f08f67d9605fe1c490d226715df1d07c10db96a563c667e6 WatchSource:0}: Error finding container 9abf7babc3a8bc84f08f67d9605fe1c490d226715df1d07c10db96a563c667e6: Status 404 returned error can't find the container with id 9abf7babc3a8bc84f08f67d9605fe1c490d226715df1d07c10db96a563c667e6
Sep 17 17:34:19.032281 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:34:19.032257    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a67c51_fa23_48cc_b243_0d6ff5843cf9.slice/crio-6bbb12268220a68e40e37ddf8f6f368e3c799edad0beae021dd229ba908989d6 WatchSource:0}: Error finding container 6bbb12268220a68e40e37ddf8f6f368e3c799edad0beae021dd229ba908989d6: Status 404 returned error can't find the container with id 6bbb12268220a68e40e37ddf8f6f368e3c799edad0beae021dd229ba908989d6
Sep 17 17:34:19.033032 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:34:19.032967    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2de2127_a021_4e72_b532_29117efb2909.slice/crio-5fa29c7fd655d18bd1ddcece8269404dd994cbbe5d2969eb756a95cf2cd897a7 WatchSource:0}: Error finding container 5fa29c7fd655d18bd1ddcece8269404dd994cbbe5d2969eb756a95cf2cd897a7: Status 404 returned error can't find the container with id 5fa29c7fd655d18bd1ddcece8269404dd994cbbe5d2969eb756a95cf2cd897a7
Sep 17 17:34:19.041503 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.041483    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9hzc9"]
Sep 17 17:34:19.044438 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:34:19.044414    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaad7b8f_2beb_46aa_91d7_cafffb2e11e4.slice/crio-8d131d4cf656ba26ecc9cb183e0ed9dadb88983dbbf5a89ad9a85a4f65f016de WatchSource:0}: Error finding container 8d131d4cf656ba26ecc9cb183e0ed9dadb88983dbbf5a89ad9a85a4f65f016de: Status 404 returned error can't find the container with id 8d131d4cf656ba26ecc9cb183e0ed9dadb88983dbbf5a89ad9a85a4f65f016de
Sep 17 17:34:19.125602 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.125576    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:19.125961 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:19.125721    2521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:19.125961 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:19.125774    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls podName:058158ca-fba0-4467-9c85-4661b64588ee nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.125760338 +0000 UTC m=+35.170765425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls") pod "dns-default-6zpfd" (UID: "058158ca-fba0-4467-9c85-4661b64588ee") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:19.793301 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.793179    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf" event={"ID":"5252e4c9-2542-43d0-9262-fd67eb5eea6d","Type":"ContainerStarted","Data":"9abf7babc3a8bc84f08f67d9605fe1c490d226715df1d07c10db96a563c667e6"}
Sep 17 17:34:19.795033 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.794961    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r" event={"ID":"c3a67c51-fa23-48cc-b243-0d6ff5843cf9","Type":"ContainerStarted","Data":"6bbb12268220a68e40e37ddf8f6f368e3c799edad0beae021dd229ba908989d6"}
Sep 17 17:34:19.799382 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.799353    2521 generic.go:356] "Generic (PLEG): container finished" podID="8c529cb3-09aa-42d0-ae31-9b76df0ab375" containerID="6dee92386e68b619bf03afca285544152ce5509312e76eaa31bd67cdac55be1e" exitCode=0
Sep 17 17:34:19.799515 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.799445    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9zzms" event={"ID":"8c529cb3-09aa-42d0-ae31-9b76df0ab375","Type":"ContainerDied","Data":"6dee92386e68b619bf03afca285544152ce5509312e76eaa31bd67cdac55be1e"}
Sep 17 17:34:19.802212 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.802186    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9hzc9" event={"ID":"daad7b8f-2beb-46aa-91d7-cafffb2e11e4","Type":"ContainerStarted","Data":"8d131d4cf656ba26ecc9cb183e0ed9dadb88983dbbf5a89ad9a85a4f65f016de"}
Sep 17 17:34:19.804046 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:19.804021    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" event={"ID":"c2de2127-a021-4e72-b532-29117efb2909","Type":"ContainerStarted","Data":"5fa29c7fd655d18bd1ddcece8269404dd994cbbe5d2969eb756a95cf2cd897a7"}
Sep 17 17:34:20.035002 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:20.034933    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:20.035229 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:20.035120    2521 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:20.035229 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:20.035138    2521 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb59946d7-hbp89: secret "image-registry-tls" not found
Sep 17 17:34:20.035229 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:20.035198    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls podName:1b539bd7-2cb1-4894-acd7-0579cf3b5bc1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:22.035181172 +0000 UTC m=+37.080186239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls") pod "image-registry-6bb59946d7-hbp89" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1") : secret "image-registry-tls" not found
Sep 17 17:34:20.135956 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:20.135768    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:20.136553 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:20.135959    2521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:20.136553 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:20.136026    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls podName:058158ca-fba0-4467-9c85-4661b64588ee nodeName:}" failed. No retries permitted until 2026-09-17 17:34:22.136009126 +0000 UTC m=+37.181014193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls") pod "dns-default-6zpfd" (UID: "058158ca-fba0-4467-9c85-4661b64588ee") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:20.816649 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:20.816571    2521 generic.go:356] "Generic (PLEG): container finished" podID="8c529cb3-09aa-42d0-ae31-9b76df0ab375" containerID="cbbd3b927c01671aa120235f49ae071615c10b65a392fb3c2c32a9a164020114" exitCode=0
Sep 17 17:34:20.816649 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:20.816615    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9zzms" event={"ID":"8c529cb3-09aa-42d0-ae31-9b76df0ab375","Type":"ContainerDied","Data":"cbbd3b927c01671aa120235f49ae071615c10b65a392fb3c2c32a9a164020114"}
Sep 17 17:34:22.055387 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:22.055341    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:22.056039 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:22.055516    2521 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:22.056039 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:22.055530    2521 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb59946d7-hbp89: secret "image-registry-tls" not found
Sep 17 17:34:22.056039 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:22.055603    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls podName:1b539bd7-2cb1-4894-acd7-0579cf3b5bc1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:26.055586878 +0000 UTC m=+41.100591952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls") pod "image-registry-6bb59946d7-hbp89" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1") : secret "image-registry-tls" not found
Sep 17 17:34:22.156586 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:22.156454    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:22.156917 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:22.156849    2521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:22.157020 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:22.156933    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls podName:058158ca-fba0-4467-9c85-4661b64588ee nodeName:}" failed. No retries permitted until 2026-09-17 17:34:26.156909686 +0000 UTC m=+41.201914760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls") pod "dns-default-6zpfd" (UID: "058158ca-fba0-4467-9c85-4661b64588ee") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:26.091122 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.091079    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:26.091597 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:26.091239    2521 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:26.091597 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:26.091258    2521 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb59946d7-hbp89: secret "image-registry-tls" not found
Sep 17 17:34:26.091597 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:26.091326    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls podName:1b539bd7-2cb1-4894-acd7-0579cf3b5bc1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:34.091308154 +0000 UTC m=+49.136313221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls") pod "image-registry-6bb59946d7-hbp89" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1") : secret "image-registry-tls" not found
Sep 17 17:34:26.192424 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.192383    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:26.192672 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:26.192574    2521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:26.192672 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:26.192659    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls podName:058158ca-fba0-4467-9c85-4661b64588ee nodeName:}" failed. No retries permitted until 2026-09-17 17:34:34.192637564 +0000 UTC m=+49.237642649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls") pod "dns-default-6zpfd" (UID: "058158ca-fba0-4467-9c85-4661b64588ee") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:26.835385 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.835306    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9hzc9" event={"ID":"daad7b8f-2beb-46aa-91d7-cafffb2e11e4","Type":"ContainerStarted","Data":"f0a0de1565ea7fea628474d1a437f857351db325e58e757cd0494a05b299c50c"}
Sep 17 17:34:26.837144 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.837105    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" event={"ID":"c2de2127-a021-4e72-b532-29117efb2909","Type":"ContainerStarted","Data":"abb17164e3d08bfa074768857bc8705e030bff4836bc16d2ab0dfd594895fbbc"}
Sep 17 17:34:26.839108 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.838940    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r" event={"ID":"c3a67c51-fa23-48cc-b243-0d6ff5843cf9","Type":"ContainerStarted","Data":"3b87cb262a15ed4b77e2bf5ae3c4ad8e27d4652262d904a5951ee5fc7d0bbf48"}
Sep 17 17:34:26.841788 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.841739    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9zzms" event={"ID":"8c529cb3-09aa-42d0-ae31-9b76df0ab375","Type":"ContainerStarted","Data":"cae1d3f7a50afd575d9f042259e569264f7789960182e63d934cd639e2532da0"}
Sep 17 17:34:26.849511 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.849328    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:26.850624 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.850605    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:34:26.867515 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.867417    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9hzc9" podStartSLOduration=1.292184803 podStartE2EDuration="8.867398451s" totalImagesPullingTime="7.575213648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:26.86710995 +0000 UTC m=+41.912115236" watchObservedRunningTime="2026-09-17 17:34:26.867398451 +0000 UTC m=+41.912403542"
Sep 17 17:34:26.895575 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.895500    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9zzms" podStartSLOduration=13.209244809 podStartE2EDuration="41.895479548s" totalImagesPullingTime="28.686234739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:26.893762336 +0000 UTC m=+41.938767425" watchObservedRunningTime="2026-09-17 17:34:26.895479548 +0000 UTC m=+41.940484641"
Sep 17 17:34:26.916463 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:26.916400    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r" podStartSLOduration=12.277031662 podStartE2EDuration="19.916378431s" totalImagesPullingTime="7.639346769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:26.91522633 +0000 UTC m=+41.960231420" watchObservedRunningTime="2026-09-17 17:34:26.916378431 +0000 UTC m=+41.961383520"
Sep 17 17:34:27.846986 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:27.846951    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" event={"ID":"c2de2127-a021-4e72-b532-29117efb2909","Type":"ContainerStarted","Data":"d022c6f26719b135753ffe6a3550247357ab8302e4a104db6683e0827e3db2ff"}
Sep 17 17:34:27.847373 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:27.847006    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" event={"ID":"c2de2127-a021-4e72-b532-29117efb2909","Type":"ContainerStarted","Data":"b00abc21f8556decae142e3211699f7e13799808803dbd6da9ea229b91874bab"}
Sep 17 17:34:27.879446 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:27.879381    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" podStartSLOduration=13.225771719 podStartE2EDuration="20.879363524s" totalImagesPullingTime="7.653591805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:27.878681682 +0000 UTC m=+42.923686771" watchObservedRunningTime="2026-09-17 17:34:27.879363524 +0000 UTC m=+42.924368614"
Sep 17 17:34:30.857104 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:30.857069    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf" event={"ID":"5252e4c9-2542-43d0-9262-fd67eb5eea6d","Type":"ContainerStarted","Data":"356d1535d8ecdeb06aa78fb8cb29e6e2496fde347ac8b2ced4f318977292c0b5"}
Sep 17 17:34:34.157299 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:34.157256    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:34.157712 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:34.157404    2521 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:34.157712 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:34.157422    2521 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb59946d7-hbp89: secret "image-registry-tls" not found
Sep 17 17:34:34.157712 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:34.157475    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls podName:1b539bd7-2cb1-4894-acd7-0579cf3b5bc1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.157461179 +0000 UTC m=+65.202466246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls") pod "image-registry-6bb59946d7-hbp89" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1") : secret "image-registry-tls" not found
Sep 17 17:34:34.257931 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:34.257889    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:34.258118 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:34.258054    2521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:34.258160 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:34.258126    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls podName:058158ca-fba0-4467-9c85-4661b64588ee nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.258110453 +0000 UTC m=+65.303115520 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls") pod "dns-default-6zpfd" (UID: "058158ca-fba0-4467-9c85-4661b64588ee") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:44.787180 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:44.787152    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9qk8j"
Sep 17 17:34:44.820153 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:44.820099    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf" podStartSLOduration=26.326289134 podStartE2EDuration="37.820082495s" totalImagesPullingTime="11.493793361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:30.876626322 +0000 UTC m=+45.921631421" watchObservedRunningTime="2026-09-17 17:34:44.820082495 +0000 UTC m=+59.865087584"
Sep 17 17:34:50.172815 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.172769    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:34:50.173410 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.172841    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:34:50.173410 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:50.172972    2521 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:50.173410 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:50.172989    2521 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb59946d7-hbp89: secret "image-registry-tls" not found
Sep 17 17:34:50.173410 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:50.173063    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls podName:1b539bd7-2cb1-4894-acd7-0579cf3b5bc1 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:22.173043388 +0000 UTC m=+97.218048456 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls") pod "image-registry-6bb59946d7-hbp89" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1") : secret "image-registry-tls" not found
Sep 17 17:34:50.175904 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.175884    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:50.183579 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:50.183543    2521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:34:50.183689 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:50.183648    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs podName:e8594596-9c6f-4e0f-9892-21e06eba0cad nodeName:}" failed. No retries permitted until 2026-09-17 17:35:54.183627467 +0000 UTC m=+129.228632544 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs") pod "network-metrics-daemon-n6dxl" (UID: "e8594596-9c6f-4e0f-9892-21e06eba0cad") : secret "metrics-daemon-secret" not found
Sep 17 17:34:50.273527 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.273489    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:34:50.273732 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:50.273654    2521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:50.273732 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.273682    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:50.273732 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:34:50.273709    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls podName:058158ca-fba0-4467-9c85-4661b64588ee nodeName:}" failed. No retries permitted until 2026-09-17 17:35:22.273693931 +0000 UTC m=+97.318699002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls") pod "dns-default-6zpfd" (UID: "058158ca-fba0-4467-9c85-4661b64588ee") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:50.276431 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.276413    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:50.287723 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.287703    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2da9dc7f-e3f7-442b-bf4e-badb16e819bd-original-pull-secret\") pod \"global-pull-secret-syncer-vzqnh\" (UID: \"2da9dc7f-e3f7-442b-bf4e-badb16e819bd\") " pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:50.374778 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.374738    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:50.377610 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.377590    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:50.388035 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.388012    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:50.398472 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.398449    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrdr\" (UniqueName: \"kubernetes.io/projected/3e16cb48-397c-45e8-94c0-e40e16b52d8c-kube-api-access-qdrdr\") pod \"network-check-target-h4vpg\" (UID: \"3e16cb48-397c-45e8-94c0-e40e16b52d8c\") " pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:50.581231 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.581143    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vzqnh"
Sep 17 17:34:50.592195 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.592165    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7zxgv\""
Sep 17 17:34:50.599619 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.599590    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:50.884230 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.884158    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vzqnh"]
Sep 17 17:34:50.887093 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:34:50.887062    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da9dc7f_e3f7_442b_bf4e_badb16e819bd.slice/crio-219cae7eed669cc75fd21a28a291c965fe8d70d822fc1c56147383d268b35bc9 WatchSource:0}: Error finding container 219cae7eed669cc75fd21a28a291c965fe8d70d822fc1c56147383d268b35bc9: Status 404 returned error can't find the container with id 219cae7eed669cc75fd21a28a291c965fe8d70d822fc1c56147383d268b35bc9
Sep 17 17:34:50.910692 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.910668    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4vpg"]
Sep 17 17:34:50.913727 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:34:50.913699    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e16cb48_397c_45e8_94c0_e40e16b52d8c.slice/crio-5ec6a0ffd393ea56ade517ebf0ce6f904eddf3bb83458db069103e41dbc423e4 WatchSource:0}: Error finding container 5ec6a0ffd393ea56ade517ebf0ce6f904eddf3bb83458db069103e41dbc423e4: Status 404 returned error can't find the container with id 5ec6a0ffd393ea56ade517ebf0ce6f904eddf3bb83458db069103e41dbc423e4
Sep 17 17:34:50.922076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.922047    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vzqnh" event={"ID":"2da9dc7f-e3f7-442b-bf4e-badb16e819bd","Type":"ContainerStarted","Data":"219cae7eed669cc75fd21a28a291c965fe8d70d822fc1c56147383d268b35bc9"}
Sep 17 17:34:50.923206 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:50.923174    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4vpg" event={"ID":"3e16cb48-397c-45e8-94c0-e40e16b52d8c","Type":"ContainerStarted","Data":"5ec6a0ffd393ea56ade517ebf0ce6f904eddf3bb83458db069103e41dbc423e4"}
Sep 17 17:34:55.943121 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:55.943079    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vzqnh" event={"ID":"2da9dc7f-e3f7-442b-bf4e-badb16e819bd","Type":"ContainerStarted","Data":"b7ae8936d7ab474095a0ae50f773b2dc161cd4bb497c355784b63bb85bf919d5"}
Sep 17 17:34:55.944534 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:55.944510    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4vpg" event={"ID":"3e16cb48-397c-45e8-94c0-e40e16b52d8c","Type":"ContainerStarted","Data":"c3b97600405b8633433126a878c47bcdaec49fd0ce2dc67c07d5106ea2f109be"}
Sep 17 17:34:55.944620 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:55.944586    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:34:55.961616 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:55.961547    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vzqnh" podStartSLOduration=65.64403294 podStartE2EDuration="1m9.961534279s" totalImagesPullingTime="4.317501339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:55.9605815 +0000 UTC m=+71.005586580" watchObservedRunningTime="2026-09-17 17:34:55.961534279 +0000 UTC m=+71.006539368"
Sep 17 17:34:55.982869 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:34:55.982826    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h4vpg" podStartSLOduration=66.696879711 podStartE2EDuration="1m10.982812552s" totalImagesPullingTime="4.285932841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:55.982323939 +0000 UTC m=+71.027329027" watchObservedRunningTime="2026-09-17 17:34:55.982812552 +0000 UTC m=+71.027817638"
Sep 17 17:35:22.210213 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:35:22.210181    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:35:22.210710 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:35:22.210334    2521 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:22.210710 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:35:22.210350    2521 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb59946d7-hbp89: secret "image-registry-tls" not found
Sep 17 17:35:22.210710 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:35:22.210406    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls podName:1b539bd7-2cb1-4894-acd7-0579cf3b5bc1 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.210390079 +0000 UTC m=+161.255395160 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls") pod "image-registry-6bb59946d7-hbp89" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1") : secret "image-registry-tls" not found
Sep 17 17:35:22.310917 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:35:22.310869    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:35:22.311037 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:35:22.311018    2521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:22.311099 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:35:22.311090    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls podName:058158ca-fba0-4467-9c85-4661b64588ee nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.311075823 +0000 UTC m=+161.356080894 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls") pod "dns-default-6zpfd" (UID: "058158ca-fba0-4467-9c85-4661b64588ee") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:26.950263 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:35:26.950231    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h4vpg"
Sep 17 17:35:54.245256 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:35:54.245209    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:35:54.245766 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:35:54.245362    2521 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:35:54.245766 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:35:54.245439    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs podName:e8594596-9c6f-4e0f-9892-21e06eba0cad nodeName:}" failed. No retries permitted until 2026-09-17 17:37:56.245421756 +0000 UTC m=+251.290426824 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs") pod "network-metrics-daemon-n6dxl" (UID: "e8594596-9c6f-4e0f-9892-21e06eba0cad") : secret "metrics-daemon-secret" not found
Sep 17 17:36:21.343272 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:21.343227    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6bb59946d7-hbp89" podUID="1b539bd7-2cb1-4894-acd7-0579cf3b5bc1"
Sep 17 17:36:21.416518 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:21.416476    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6zpfd" podUID="058158ca-fba0-4467-9c85-4661b64588ee"
Sep 17 17:36:21.496059 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:21.496022    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-n6dxl" podUID="e8594596-9c6f-4e0f-9892-21e06eba0cad"
Sep 17 17:36:22.186004 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:22.185971    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:36:22.186209 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:22.185971    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:36:23.010584 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.010540    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xf4jk"]
Sep 17 17:36:23.013503 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.013486    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.015955 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.015931    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:36:23.016076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.015939    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kp8x9\""
Sep 17 17:36:23.016863 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.016844    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:36:23.016967 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.016844    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:36:23.016967 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.016875    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:36:23.026903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.026881    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xf4jk"]
Sep 17 17:36:23.051654 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.051447    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.051654 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.051505    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/796e0187-1f6d-4306-bdd4-999bb67a1a21-data-volume\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.051654 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.051581    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.051654 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.051612    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z429j\" (UniqueName: \"kubernetes.io/projected/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-api-access-z429j\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.051976 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.051708    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/796e0187-1f6d-4306-bdd4-999bb67a1a21-crio-socket\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.152649 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.152539    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/796e0187-1f6d-4306-bdd4-999bb67a1a21-data-volume\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.152836 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.152714    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.152836 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.152744    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z429j\" (UniqueName: \"kubernetes.io/projected/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-api-access-z429j\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.152836 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.152796    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/796e0187-1f6d-4306-bdd4-999bb67a1a21-crio-socket\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.152985 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.152859    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.152985 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.152884    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/796e0187-1f6d-4306-bdd4-999bb67a1a21-crio-socket\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.152985 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:23.152889    2521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:23.152985 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.152933    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/796e0187-1f6d-4306-bdd4-999bb67a1a21-data-volume\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.152985 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:23.152968    2521 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.152985 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:23.152979    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:23.652958458 +0000 UTC m=+158.697963532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:23.153197 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:23.153031    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:23.653016802 +0000 UTC m=+158.698021894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.164919 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.164875    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z429j\" (UniqueName: \"kubernetes.io/projected/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-api-access-z429j\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.656739 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.656711    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.656911 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:23.656772    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:23.656911 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:23.656836    2521 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.656911 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:23.656870    2521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:23.656911 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:23.656898    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:24.656882943 +0000 UTC m=+159.701888015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:23.657053 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:23.656922    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:24.656910956 +0000 UTC m=+159.701916023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:24.665295 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:24.665241    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:24.665771 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:24.665332    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:24.665771 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:24.665378    2521 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:24.665771 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:24.665427    2521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:24.665771 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:24.665462    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.66544511 +0000 UTC m=+161.710450181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:24.665771 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:24.665478    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.665472298 +0000 UTC m=+161.710477366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:24.748092 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:24.748068    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-89wfr_3995ed8a-03ec-4c9a-b39f-014901ab9c47/dns-node-resolver/0.log"
Sep 17 17:36:25.748424 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:25.748400    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2rxr8_a18f3ab3-0d68-49b0-8189-1aa420dc674d/node-ca/0.log"
Sep 17 17:36:26.276781 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:26.276733    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") pod \"image-registry-6bb59946d7-hbp89\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") " pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:36:26.276971 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:26.276861    2521 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:26.276971 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:26.276873    2521 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb59946d7-hbp89: secret "image-registry-tls" not found
Sep 17 17:36:26.276971 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:26.276921    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls podName:1b539bd7-2cb1-4894-acd7-0579cf3b5bc1 nodeName:}" failed. No retries permitted until 2026-09-17 17:38:28.276907464 +0000 UTC m=+283.321912531 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls") pod "image-registry-6bb59946d7-hbp89" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1") : secret "image-registry-tls" not found
Sep 17 17:36:26.377670 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:26.377623    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:36:26.377899 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:26.377782    2521 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:26.377899 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:26.377842    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls podName:058158ca-fba0-4467-9c85-4661b64588ee nodeName:}" failed. No retries permitted until 2026-09-17 17:38:28.377827946 +0000 UTC m=+283.422833012 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls") pod "dns-default-6zpfd" (UID: "058158ca-fba0-4467-9c85-4661b64588ee") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:26.680439 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:26.680397    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:26.680635 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:26.680475    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:26.680635 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:26.680532    2521 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:26.680635 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:26.680576    2521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:26.680635 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:26.680621    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:30.680606591 +0000 UTC m=+165.725611658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:26.680635 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:26.680637    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:30.680630335 +0000 UTC m=+165.725635402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:26.749052 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:26.749029    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9hzc9_daad7b8f-2beb-46aa-91d7-cafffb2e11e4/serve-healthcheck-canary/0.log"
Sep 17 17:36:26.850682 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:26.850639    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r" podUID="c3a67c51-fa23-48cc-b243-0d6ff5843cf9" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/readyz\": dial tcp 10.132.0.8:8000: connect: connection refused"
Sep 17 17:36:27.200984 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:27.200952    2521 generic.go:356] "Generic (PLEG): container finished" podID="c3a67c51-fa23-48cc-b243-0d6ff5843cf9" containerID="3b87cb262a15ed4b77e2bf5ae3c4ad8e27d4652262d904a5951ee5fc7d0bbf48" exitCode=1
Sep 17 17:36:27.201154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:27.201009    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r" event={"ID":"c3a67c51-fa23-48cc-b243-0d6ff5843cf9","Type":"ContainerDied","Data":"3b87cb262a15ed4b77e2bf5ae3c4ad8e27d4652262d904a5951ee5fc7d0bbf48"}
Sep 17 17:36:27.204740 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:27.201683    2521 scope.go:117] "RemoveContainer" containerID="3b87cb262a15ed4b77e2bf5ae3c4ad8e27d4652262d904a5951ee5fc7d0bbf48"
Sep 17 17:36:28.206212 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:28.206175    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r" event={"ID":"c3a67c51-fa23-48cc-b243-0d6ff5843cf9","Type":"ContainerStarted","Data":"25b4756e846dbd67b074a68343993478772ddf5b9d4aadb28b3d7849b352f55a"}
Sep 17 17:36:28.206618 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:28.206490    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:36:28.207088 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:28.207071    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f48bcb4-vwv5r"
Sep 17 17:36:30.711688 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:30.711652    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:30.712085 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:30.711717    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:30.712085 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:30.711800    2521 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:30.712085 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:30.711838    2521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:30.712085 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:30.711859    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:38.711843536 +0000 UTC m=+173.756848615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:30.712085 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:30.711904    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:38.711885162 +0000 UTC m=+173.756890229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:31.217500 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:31.217464    2521 generic.go:356] "Generic (PLEG): container finished" podID="5252e4c9-2542-43d0-9262-fd67eb5eea6d" containerID="356d1535d8ecdeb06aa78fb8cb29e6e2496fde347ac8b2ced4f318977292c0b5" exitCode=255
Sep 17 17:36:31.217710 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:31.217519    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf" event={"ID":"5252e4c9-2542-43d0-9262-fd67eb5eea6d","Type":"ContainerDied","Data":"356d1535d8ecdeb06aa78fb8cb29e6e2496fde347ac8b2ced4f318977292c0b5"}
Sep 17 17:36:31.217906 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:31.217888    2521 scope.go:117] "RemoveContainer" containerID="356d1535d8ecdeb06aa78fb8cb29e6e2496fde347ac8b2ced4f318977292c0b5"
Sep 17 17:36:32.222516 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:32.222478    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6548c74dcb-ntxjf" event={"ID":"5252e4c9-2542-43d0-9262-fd67eb5eea6d","Type":"ContainerStarted","Data":"ad9faf0ce56b0befc0db376b37595e79e43d0832f8bacfd74057f4668363dd75"}
Sep 17 17:36:35.469913 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:35.469883    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:36:38.772846 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:38.772742    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:38.772846 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:38.772822    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:38.773257 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:38.772923    2521 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:38.773257 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:36:38.772975    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls podName:796e0187-1f6d-4306-bdd4-999bb67a1a21 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:54.772962143 +0000 UTC m=+189.817967210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xf4jk" (UID: "796e0187-1f6d-4306-bdd4-999bb67a1a21") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:38.773333 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:38.773284    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/796e0187-1f6d-4306-bdd4-999bb67a1a21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:48.683266 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:48.683230    2521 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" podUID="c2de2127-a021-4e72-b532-29117efb2909" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:36:54.793612 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:54.793542    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:54.795936 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:54.795911    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/796e0187-1f6d-4306-bdd4-999bb67a1a21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xf4jk\" (UID: \"796e0187-1f6d-4306-bdd4-999bb67a1a21\") " pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:54.825675 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:54.825638    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kp8x9\""
Sep 17 17:36:54.833673 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:54.833641    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xf4jk"
Sep 17 17:36:55.103162 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:55.102996    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xf4jk"]
Sep 17 17:36:55.105448 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:36:55.105419    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796e0187_1f6d_4306_bdd4_999bb67a1a21.slice/crio-f1ced72ec8024bcb33d14530a169a2c523907fc468bab3d1891056e8675d8a38 WatchSource:0}: Error finding container f1ced72ec8024bcb33d14530a169a2c523907fc468bab3d1891056e8675d8a38: Status 404 returned error can't find the container with id f1ced72ec8024bcb33d14530a169a2c523907fc468bab3d1891056e8675d8a38
Sep 17 17:36:55.293102 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:55.293059    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xf4jk" event={"ID":"796e0187-1f6d-4306-bdd4-999bb67a1a21","Type":"ContainerStarted","Data":"963277a7e1976cdf4b24a6eb9fdb7c4b32e40a474e5750cc48aaa7c78f8803b8"}
Sep 17 17:36:55.293283 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:55.293125    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xf4jk" event={"ID":"796e0187-1f6d-4306-bdd4-999bb67a1a21","Type":"ContainerStarted","Data":"f1ced72ec8024bcb33d14530a169a2c523907fc468bab3d1891056e8675d8a38"}
Sep 17 17:36:56.298648 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:56.298613    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xf4jk" event={"ID":"796e0187-1f6d-4306-bdd4-999bb67a1a21","Type":"ContainerStarted","Data":"b4b1138cf5b961b11d8795ae42104cdeda199b92c5a80d278c0708b6d3a4e8d5"}
Sep 17 17:36:57.304209 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:57.304121    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xf4jk" event={"ID":"796e0187-1f6d-4306-bdd4-999bb67a1a21","Type":"ContainerStarted","Data":"773476314a8f0d3a92452dec1e7cc38e1d05afbf2c00479a7553371c4367d3f6"}
Sep 17 17:36:57.326624 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:57.326553    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xf4jk" podStartSLOduration=33.4855421 podStartE2EDuration="35.326536716s" totalImagesPullingTime="1.840994616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:36:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:57.325912901 +0000 UTC m=+192.370917990" watchObservedRunningTime="2026-09-17 17:36:57.326536716 +0000 UTC m=+192.371541804"
Sep 17 17:36:58.683612 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:36:58.683574    2521 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" podUID="c2de2127-a021-4e72-b532-29117efb2909" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:37:08.683869 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:08.683826    2521 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" podUID="c2de2127-a021-4e72-b532-29117efb2909" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:37:08.684224 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:08.683924    2521 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp"
Sep 17 17:37:08.684493 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:08.684462    2521 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"d022c6f26719b135753ffe6a3550247357ab8302e4a104db6683e0827e3db2ff"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 17:37:08.684591 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:08.684517    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" podUID="c2de2127-a021-4e72-b532-29117efb2909" containerName="service-proxy" containerID="cri-o://d022c6f26719b135753ffe6a3550247357ab8302e4a104db6683e0827e3db2ff" gracePeriod=30
Sep 17 17:37:09.342259 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.342223    2521 generic.go:356] "Generic (PLEG): container finished" podID="c2de2127-a021-4e72-b532-29117efb2909" containerID="d022c6f26719b135753ffe6a3550247357ab8302e4a104db6683e0827e3db2ff" exitCode=2
Sep 17 17:37:09.342259 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.342262    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" event={"ID":"c2de2127-a021-4e72-b532-29117efb2909","Type":"ContainerDied","Data":"d022c6f26719b135753ffe6a3550247357ab8302e4a104db6683e0827e3db2ff"}
Sep 17 17:37:09.342466 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.342295    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b8d87dcc6-7dbsp" event={"ID":"c2de2127-a021-4e72-b532-29117efb2909","Type":"ContainerStarted","Data":"f4df388ddccd40af19e58258edc4311eb7bee91f2726c5b862d81e489d2b519e"}
Sep 17 17:37:09.490174 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.490137    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bb59946d7-hbp89"]
Sep 17 17:37:09.490393 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:37:09.490374    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6bb59946d7-hbp89" podUID="1b539bd7-2cb1-4894-acd7-0579cf3b5bc1"
Sep 17 17:37:09.645767 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.645736    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8drvh"]
Sep 17 17:37:09.648886 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.648870    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.651288 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.651269    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:37:09.653725 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.653703    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:37:09.653823 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.653746    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:37:09.653879 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.653836    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:37:09.654537 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.654517    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dq864\""
Sep 17 17:37:09.654663 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.654541    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:37:09.654663 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.654592    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:37:09.813690 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.813650    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.814076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.813734    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptgt6\" (UniqueName: \"kubernetes.io/projected/7821b4dd-48eb-4488-9761-b3b27be661f4-kube-api-access-ptgt6\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.814076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.813755    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7821b4dd-48eb-4488-9761-b3b27be661f4-metrics-client-ca\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.814076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.813776    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-textfile\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.814076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.813791    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7821b4dd-48eb-4488-9761-b3b27be661f4-sys\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.814076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.813807    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7821b4dd-48eb-4488-9761-b3b27be661f4-root\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.814076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.813834    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-wtmp\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.814076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.813850    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-tls\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.814076 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.813893    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.914897 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.914817    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-textfile\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.914897 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.914848    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7821b4dd-48eb-4488-9761-b3b27be661f4-sys\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.914897 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.914866    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7821b4dd-48eb-4488-9761-b3b27be661f4-root\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.914897 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.914884    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-wtmp\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.914897 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.914902    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-tls\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.914949    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7821b4dd-48eb-4488-9761-b3b27be661f4-sys\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.914967    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7821b4dd-48eb-4488-9761-b3b27be661f4-root\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.914956    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.915032    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.915051    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-wtmp\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.915124    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptgt6\" (UniqueName: \"kubernetes.io/projected/7821b4dd-48eb-4488-9761-b3b27be661f4-kube-api-access-ptgt6\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.915146    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7821b4dd-48eb-4488-9761-b3b27be661f4-metrics-client-ca\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915255 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.915257    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-textfile\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915607 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.915586    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.915654 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.915638    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7821b4dd-48eb-4488-9761-b3b27be661f4-metrics-client-ca\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.917206 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.917185    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.917331 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.917307    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7821b4dd-48eb-4488-9761-b3b27be661f4-node-exporter-tls\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:09.960692 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:09.960665    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptgt6\" (UniqueName: \"kubernetes.io/projected/7821b4dd-48eb-4488-9761-b3b27be661f4-kube-api-access-ptgt6\") pod \"node-exporter-8drvh\" (UID: \"7821b4dd-48eb-4488-9761-b3b27be661f4\") " pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:10.257671 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.257550    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8drvh"
Sep 17 17:37:10.268907 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:37:10.268878    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7821b4dd_48eb_4488_9761_b3b27be661f4.slice/crio-5730d79d0c01d6fc4834fc5fb4657fcbe5683e3a09f1e775667dd37411a42493 WatchSource:0}: Error finding container 5730d79d0c01d6fc4834fc5fb4657fcbe5683e3a09f1e775667dd37411a42493: Status 404 returned error can't find the container with id 5730d79d0c01d6fc4834fc5fb4657fcbe5683e3a09f1e775667dd37411a42493
Sep 17 17:37:10.347225 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.347189    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8drvh" event={"ID":"7821b4dd-48eb-4488-9761-b3b27be661f4","Type":"ContainerStarted","Data":"5730d79d0c01d6fc4834fc5fb4657fcbe5683e3a09f1e775667dd37411a42493"}
Sep 17 17:37:10.347225 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.347221    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:37:10.354377 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.354347    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:37:10.519619 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.519511    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-certificates\") pod \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") "
Sep 17 17:37:10.519619 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.519588    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-ca-trust-extracted\") pod \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") "
Sep 17 17:37:10.519793 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.519700    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-trusted-ca\") pod \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") "
Sep 17 17:37:10.519793 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.519744    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-bound-sa-token\") pod \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") "
Sep 17 17:37:10.519793 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.519778    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-installation-pull-secrets\") pod \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") "
Sep 17 17:37:10.519915 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.519827    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdm7w\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-kube-api-access-pdm7w\") pod \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") "
Sep 17 17:37:10.519915 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.519858    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-image-registry-private-configuration\") pod \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\" (UID: \"1b539bd7-2cb1-4894-acd7-0579cf3b5bc1\") "
Sep 17 17:37:10.519915 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.519825    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:10.520059 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.519932    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:10.520149 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.520126    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:10.520254 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.520231    2521 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-certificates\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:37:10.520254 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.520253    2521 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-ca-trust-extracted\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:37:10.520375 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.520267    2521 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-trusted-ca\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:37:10.522199 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.522173    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:10.522199 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.522190    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:10.522338 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.522243    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-kube-api-access-pdm7w" (OuterVolumeSpecName: "kube-api-access-pdm7w") pod "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1"). InnerVolumeSpecName "kube-api-access-pdm7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:10.522338 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.522250    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1" (UID: "1b539bd7-2cb1-4894-acd7-0579cf3b5bc1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:10.621090 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.621035    2521 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-bound-sa-token\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:37:10.621090 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.621066    2521 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-installation-pull-secrets\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:37:10.621090 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.621082    2521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdm7w\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-kube-api-access-pdm7w\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:37:10.621090 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:10.621096    2521 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-image-registry-private-configuration\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:37:11.352279 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:11.352242    2521 generic.go:356] "Generic (PLEG): container finished" podID="7821b4dd-48eb-4488-9761-b3b27be661f4" containerID="f2de0e9312334052733858e5b4ac67cce761d7ebf9e137cbfeb4b5b8337d4d4e" exitCode=0
Sep 17 17:37:11.352651 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:11.352323    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8drvh" event={"ID":"7821b4dd-48eb-4488-9761-b3b27be661f4","Type":"ContainerDied","Data":"f2de0e9312334052733858e5b4ac67cce761d7ebf9e137cbfeb4b5b8337d4d4e"}
Sep 17 17:37:11.352651 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:11.352360    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb59946d7-hbp89"
Sep 17 17:37:11.440014 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:11.439987    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bb59946d7-hbp89"]
Sep 17 17:37:11.450865 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:11.450834    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bb59946d7-hbp89"]
Sep 17 17:37:11.473442 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:11.473414    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b539bd7-2cb1-4894-acd7-0579cf3b5bc1" path="/var/lib/kubelet/pods/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1/volumes"
Sep 17 17:37:11.629070 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:11.629034    2521 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b539bd7-2cb1-4894-acd7-0579cf3b5bc1-registry-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:37:12.357285 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:12.357243    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8drvh" event={"ID":"7821b4dd-48eb-4488-9761-b3b27be661f4","Type":"ContainerStarted","Data":"3ab3825db8ad4493131491c822e46373d9c0c0a0f7995309d84918aac76f5e67"}
Sep 17 17:37:12.357703 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:12.357310    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8drvh" event={"ID":"7821b4dd-48eb-4488-9761-b3b27be661f4","Type":"ContainerStarted","Data":"6fe25219176705a1bad1d6a537907634c200b3a1b7f1de542d9cfd3c682fbbd2"}
Sep 17 17:37:12.385724 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:12.385668    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8drvh" podStartSLOduration=2.453933523 podStartE2EDuration="3.385644176s" totalImagesPullingTime="870.814424ms" totalInitContainerRuntime="60.896229ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:12.385198987 +0000 UTC m=+207.430204077" watchObservedRunningTime="2026-09-17 17:37:12.385644176 +0000 UTC m=+207.430649269"
Sep 17 17:37:53.735583 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:53.735536    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-89wfr_3995ed8a-03ec-4c9a-b39f-014901ab9c47/dns-node-resolver/0.log"
Sep 17 17:37:56.252661 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:56.252608    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:37:56.254980 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:56.254960    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8594596-9c6f-4e0f-9892-21e06eba0cad-metrics-certs\") pod \"network-metrics-daemon-n6dxl\" (UID: \"e8594596-9c6f-4e0f-9892-21e06eba0cad\") " pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:37:56.473971 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:56.473941    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-745tx\""
Sep 17 17:37:56.482112 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:56.482097    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n6dxl"
Sep 17 17:37:56.751739 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:56.751552    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n6dxl"]
Sep 17 17:37:56.754215 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:37:56.754186    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8594596_9c6f_4e0f_9892_21e06eba0cad.slice/crio-79c76f136341916ed0ad594f3543e6f9b0c52eec01fc4481d8f0047c72215e13 WatchSource:0}: Error finding container 79c76f136341916ed0ad594f3543e6f9b0c52eec01fc4481d8f0047c72215e13: Status 404 returned error can't find the container with id 79c76f136341916ed0ad594f3543e6f9b0c52eec01fc4481d8f0047c72215e13
Sep 17 17:37:57.496411 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:57.496369    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n6dxl" event={"ID":"e8594596-9c6f-4e0f-9892-21e06eba0cad","Type":"ContainerStarted","Data":"79c76f136341916ed0ad594f3543e6f9b0c52eec01fc4481d8f0047c72215e13"}
Sep 17 17:37:58.501630 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:58.501589    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n6dxl" event={"ID":"e8594596-9c6f-4e0f-9892-21e06eba0cad","Type":"ContainerStarted","Data":"7cb7ba0dad1dd0e910facf9cd12a7a3640c393896f765f400f7aff542dbb8bad"}
Sep 17 17:37:58.501998 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:58.501652    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n6dxl" event={"ID":"e8594596-9c6f-4e0f-9892-21e06eba0cad","Type":"ContainerStarted","Data":"d3fd0dc1b085c811234fab2e6758b3271f898af344e3709976e174f72ea66768"}
Sep 17 17:37:58.524046 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:37:58.523998    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n6dxl" podStartSLOduration=252.209339323 podStartE2EDuration="4m13.523982328s" totalImagesPullingTime="1.314643005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:58.52319322 +0000 UTC m=+253.568198309" watchObservedRunningTime="2026-09-17 17:37:58.523982328 +0000 UTC m=+253.568987417"
Sep 17 17:38:25.187470 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:38:25.187419    2521 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6zpfd" podUID="058158ca-fba0-4467-9c85-4661b64588ee"
Sep 17 17:38:25.584208 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:25.584129    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:38:28.473579 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:28.473523    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:38:28.475747 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:28.475728    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/058158ca-fba0-4467-9c85-4661b64588ee-metrics-tls\") pod \"dns-default-6zpfd\" (UID: \"058158ca-fba0-4467-9c85-4661b64588ee\") " pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:38:28.587634 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:28.587596    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdp8v\""
Sep 17 17:38:28.595019 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:28.594999    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:38:28.865518 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:28.865439    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6zpfd"]
Sep 17 17:38:28.869248 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:38:28.869219    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058158ca_fba0_4467_9c85_4661b64588ee.slice/crio-07fb064f999edb20e21ff34ccdbaf96dcfcb218f020a70ced6b781943fd0cba3 WatchSource:0}: Error finding container 07fb064f999edb20e21ff34ccdbaf96dcfcb218f020a70ced6b781943fd0cba3: Status 404 returned error can't find the container with id 07fb064f999edb20e21ff34ccdbaf96dcfcb218f020a70ced6b781943fd0cba3
Sep 17 17:38:29.598708 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:29.598671    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6zpfd" event={"ID":"058158ca-fba0-4467-9c85-4661b64588ee","Type":"ContainerStarted","Data":"07fb064f999edb20e21ff34ccdbaf96dcfcb218f020a70ced6b781943fd0cba3"}
Sep 17 17:38:31.610379 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:31.610345    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6zpfd" event={"ID":"058158ca-fba0-4467-9c85-4661b64588ee","Type":"ContainerStarted","Data":"a4b142a5d70e7e11465191a477fef32d12d4c9012bc79ca4d9b4b597f4a13e8c"}
Sep 17 17:38:31.610797 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:31.610395    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6zpfd" event={"ID":"058158ca-fba0-4467-9c85-4661b64588ee","Type":"ContainerStarted","Data":"2609db0afd721fe86f8985bbc05697663de802283a81de38724d929f34436ac1"}
Sep 17 17:38:31.610797 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:31.610451    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:38:31.630130 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:31.630068    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6zpfd" podStartSLOduration=251.988887278 podStartE2EDuration="4m13.630050094s" totalImagesPullingTime="1.641162816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:31.629232252 +0000 UTC m=+286.674237341" watchObservedRunningTime="2026-09-17 17:38:31.630050094 +0000 UTC m=+286.675055191"
Sep 17 17:38:41.616173 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:41.616141    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6zpfd"
Sep 17 17:38:45.383409 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:38:45.383382    2521 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:46:13.077659 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.077625    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ph788"]
Sep 17 17:46:13.080612 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.080588    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:13.083297 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.083274    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 17:46:13.083405 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.083319    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:46:13.084350 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.084328    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-z2t8s\""
Sep 17 17:46:13.084449 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.084336    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:46:13.084504 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.084464    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"]
Sep 17 17:46:13.087576 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.087353    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:13.089548 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.089529    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-d9582\""
Sep 17 17:46:13.089637 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.089529    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 17:46:13.104617 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.104596    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ph788"]
Sep 17 17:46:13.113700 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.113676    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"]
Sep 17 17:46:13.213113 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.213078    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-cert\") pod \"kserve-controller-manager-7f654675b-ph788\" (UID: \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\") " pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:13.213113 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.213110    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkd2g\" (UniqueName: \"kubernetes.io/projected/7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b-kube-api-access-tkd2g\") pod \"llmisvc-controller-manager-645f9c8fd6-vj2g2\" (UID: \"7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:13.213343 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.213196    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-vj2g2\" (UID: \"7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:13.213343 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.213256    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf29d\" (UniqueName: \"kubernetes.io/projected/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-kube-api-access-vf29d\") pod \"kserve-controller-manager-7f654675b-ph788\" (UID: \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\") " pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:13.313741 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.313705    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-cert\") pod \"kserve-controller-manager-7f654675b-ph788\" (UID: \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\") " pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:13.313741 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.313744    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkd2g\" (UniqueName: \"kubernetes.io/projected/7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b-kube-api-access-tkd2g\") pod \"llmisvc-controller-manager-645f9c8fd6-vj2g2\" (UID: \"7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:13.313990 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.313807    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-vj2g2\" (UID: \"7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:13.313990 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.313830    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf29d\" (UniqueName: \"kubernetes.io/projected/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-kube-api-access-vf29d\") pod \"kserve-controller-manager-7f654675b-ph788\" (UID: \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\") " pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:13.313990 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:46:13.313858    2521 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 17:46:13.313990 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:46:13.313932    2521 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-cert podName:08f47e26-36cc-4c9b-a0f7-dfe4de280e96 nodeName:}" failed. No retries permitted until 2026-09-17 17:46:13.813917113 +0000 UTC m=+748.858922206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-cert") pod "kserve-controller-manager-7f654675b-ph788" (UID: "08f47e26-36cc-4c9b-a0f7-dfe4de280e96") : secret "kserve-webhook-server-cert" not found
Sep 17 17:46:13.316280 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.316252    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-vj2g2\" (UID: \"7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:13.324753 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.324719    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkd2g\" (UniqueName: \"kubernetes.io/projected/7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b-kube-api-access-tkd2g\") pod \"llmisvc-controller-manager-645f9c8fd6-vj2g2\" (UID: \"7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:13.324902 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.324790    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf29d\" (UniqueName: \"kubernetes.io/projected/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-kube-api-access-vf29d\") pod \"kserve-controller-manager-7f654675b-ph788\" (UID: \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\") " pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:13.397569 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.397537    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:13.668174 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.668146    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"]
Sep 17 17:46:13.670762 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:46:13.670730    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f25468d_fe42_4d6f_8c96_a29b3f0c1c5b.slice/crio-5696ff697df48bb4e1e7b13aee47913c46ea28a05efdc5c56962e12fc0939b6d WatchSource:0}: Error finding container 5696ff697df48bb4e1e7b13aee47913c46ea28a05efdc5c56962e12fc0939b6d: Status 404 returned error can't find the container with id 5696ff697df48bb4e1e7b13aee47913c46ea28a05efdc5c56962e12fc0939b6d
Sep 17 17:46:13.672044 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.672027    2521 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:46:13.818349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.818310    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-cert\") pod \"kserve-controller-manager-7f654675b-ph788\" (UID: \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\") " pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:13.820722 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.820704    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-cert\") pod \"kserve-controller-manager-7f654675b-ph788\" (UID: \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\") " pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:13.991635 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:13.991541    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:14.075855 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:14.075823    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2" event={"ID":"7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b","Type":"ContainerStarted","Data":"5696ff697df48bb4e1e7b13aee47913c46ea28a05efdc5c56962e12fc0939b6d"}
Sep 17 17:46:14.274737 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:14.274650    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ph788"]
Sep 17 17:46:14.277853 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:46:14.277823    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f47e26_36cc_4c9b_a0f7_dfe4de280e96.slice/crio-19031c04e5f4c7cef5251234d4bfea45abf48e9ec99326e07e328498d48ffdaa WatchSource:0}: Error finding container 19031c04e5f4c7cef5251234d4bfea45abf48e9ec99326e07e328498d48ffdaa: Status 404 returned error can't find the container with id 19031c04e5f4c7cef5251234d4bfea45abf48e9ec99326e07e328498d48ffdaa
Sep 17 17:46:15.083147 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:15.083076    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-ph788" event={"ID":"08f47e26-36cc-4c9b-a0f7-dfe4de280e96","Type":"ContainerStarted","Data":"19031c04e5f4c7cef5251234d4bfea45abf48e9ec99326e07e328498d48ffdaa"}
Sep 17 17:46:17.091935 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:17.091887    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2" event={"ID":"7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b","Type":"ContainerStarted","Data":"843ee22835c4711d8495764425eabaea32fc57d9b42281e6b9bbede47e2f9a5b"}
Sep 17 17:46:17.092333 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:17.091958    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:17.093346 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:17.093324    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-ph788" event={"ID":"08f47e26-36cc-4c9b-a0f7-dfe4de280e96","Type":"ContainerStarted","Data":"dcb7bf8cc94a30705e2471ead6bf09230df30e32d018660510089ee6e550667b"}
Sep 17 17:46:17.093455 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:17.093446    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:17.115227 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:17.115119    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2" podStartSLOduration=0.939943266 podStartE2EDuration="4.115100927s" totalImagesPullingTime="3.175157661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:17.113807592 +0000 UTC m=+752.158812682" watchObservedRunningTime="2026-09-17 17:46:17.115100927 +0000 UTC m=+752.160106018"
Sep 17 17:46:17.138596 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:17.138535    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f654675b-ph788" podStartSLOduration=1.5684312089999999 podStartE2EDuration="4.138518921s" totalImagesPullingTime="2.570087712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:17.137244936 +0000 UTC m=+752.182250028" watchObservedRunningTime="2026-09-17 17:46:17.138518921 +0000 UTC m=+752.183524010"
Sep 17 17:46:48.099113 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:48.099082    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-vj2g2"
Sep 17 17:46:48.102007 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:48.101986    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:57.004200 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.004165    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ph788"]
Sep 17 17:46:57.004652 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.004384    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7f654675b-ph788" podUID="08f47e26-36cc-4c9b-a0f7-dfe4de280e96" containerName="manager" containerID="cri-o://dcb7bf8cc94a30705e2471ead6bf09230df30e32d018660510089ee6e550667b" gracePeriod=10
Sep 17 17:46:57.039689 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.039662    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f654675b-twgwm"]
Sep 17 17:46:57.042719 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.042698    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:46:57.060592 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.060553    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-twgwm"]
Sep 17 17:46:57.116657 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.116633    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15baa177-10a8-4f00-b3b9-dcef0efacbc9-cert\") pod \"kserve-controller-manager-7f654675b-twgwm\" (UID: \"15baa177-10a8-4f00-b3b9-dcef0efacbc9\") " pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:46:57.116764 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.116710    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfqr\" (UniqueName: \"kubernetes.io/projected/15baa177-10a8-4f00-b3b9-dcef0efacbc9-kube-api-access-xkfqr\") pod \"kserve-controller-manager-7f654675b-twgwm\" (UID: \"15baa177-10a8-4f00-b3b9-dcef0efacbc9\") " pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:46:57.217191 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.217142    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15baa177-10a8-4f00-b3b9-dcef0efacbc9-cert\") pod \"kserve-controller-manager-7f654675b-twgwm\" (UID: \"15baa177-10a8-4f00-b3b9-dcef0efacbc9\") " pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:46:57.217344 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.217245    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfqr\" (UniqueName: \"kubernetes.io/projected/15baa177-10a8-4f00-b3b9-dcef0efacbc9-kube-api-access-xkfqr\") pod \"kserve-controller-manager-7f654675b-twgwm\" (UID: \"15baa177-10a8-4f00-b3b9-dcef0efacbc9\") " pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:46:57.219626 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.219593    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15baa177-10a8-4f00-b3b9-dcef0efacbc9-cert\") pod \"kserve-controller-manager-7f654675b-twgwm\" (UID: \"15baa177-10a8-4f00-b3b9-dcef0efacbc9\") " pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:46:57.224532 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.224504    2521 generic.go:356] "Generic (PLEG): container finished" podID="08f47e26-36cc-4c9b-a0f7-dfe4de280e96" containerID="dcb7bf8cc94a30705e2471ead6bf09230df30e32d018660510089ee6e550667b" exitCode=0
Sep 17 17:46:57.224686 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.224594    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-ph788" event={"ID":"08f47e26-36cc-4c9b-a0f7-dfe4de280e96","Type":"ContainerDied","Data":"dcb7bf8cc94a30705e2471ead6bf09230df30e32d018660510089ee6e550667b"}
Sep 17 17:46:57.231724 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.231704    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfqr\" (UniqueName: \"kubernetes.io/projected/15baa177-10a8-4f00-b3b9-dcef0efacbc9-kube-api-access-xkfqr\") pod \"kserve-controller-manager-7f654675b-twgwm\" (UID: \"15baa177-10a8-4f00-b3b9-dcef0efacbc9\") " pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:46:57.241368 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.241350    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:57.317860 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.317778    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-cert\") pod \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\" (UID: \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\") "
Sep 17 17:46:57.317860 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.317841    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf29d\" (UniqueName: \"kubernetes.io/projected/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-kube-api-access-vf29d\") pod \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\" (UID: \"08f47e26-36cc-4c9b-a0f7-dfe4de280e96\") "
Sep 17 17:46:57.319989 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.319953    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-kube-api-access-vf29d" (OuterVolumeSpecName: "kube-api-access-vf29d") pod "08f47e26-36cc-4c9b-a0f7-dfe4de280e96" (UID: "08f47e26-36cc-4c9b-a0f7-dfe4de280e96"). InnerVolumeSpecName "kube-api-access-vf29d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:57.320106 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.320048    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-cert" (OuterVolumeSpecName: "cert") pod "08f47e26-36cc-4c9b-a0f7-dfe4de280e96" (UID: "08f47e26-36cc-4c9b-a0f7-dfe4de280e96"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:46:57.385312 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.385276    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:46:57.419197 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.419153    2521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf29d\" (UniqueName: \"kubernetes.io/projected/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-kube-api-access-vf29d\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:46:57.419197 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.419193    2521 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08f47e26-36cc-4c9b-a0f7-dfe4de280e96-cert\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:46:57.655829 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:57.655798    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-twgwm"]
Sep 17 17:46:57.658648 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:46:57.658623    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15baa177_10a8_4f00_b3b9_dcef0efacbc9.slice/crio-e609248f4cf2c4fb2abcca95fdfd3c5f3aaba068fb8e1f2a926b4037839ea5a4 WatchSource:0}: Error finding container e609248f4cf2c4fb2abcca95fdfd3c5f3aaba068fb8e1f2a926b4037839ea5a4: Status 404 returned error can't find the container with id e609248f4cf2c4fb2abcca95fdfd3c5f3aaba068fb8e1f2a926b4037839ea5a4
Sep 17 17:46:58.230291 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:58.230250    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-ph788"
Sep 17 17:46:58.230745 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:58.230250    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-ph788" event={"ID":"08f47e26-36cc-4c9b-a0f7-dfe4de280e96","Type":"ContainerDied","Data":"19031c04e5f4c7cef5251234d4bfea45abf48e9ec99326e07e328498d48ffdaa"}
Sep 17 17:46:58.230745 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:58.230398    2521 scope.go:117] "RemoveContainer" containerID="dcb7bf8cc94a30705e2471ead6bf09230df30e32d018660510089ee6e550667b"
Sep 17 17:46:58.231962 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:58.231939    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-twgwm" event={"ID":"15baa177-10a8-4f00-b3b9-dcef0efacbc9","Type":"ContainerStarted","Data":"8676e2bdeabde2ba812251996650ac052303eb9ffd8902655b30bd940b2e0fa8"}
Sep 17 17:46:58.232058 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:58.231987    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-twgwm" event={"ID":"15baa177-10a8-4f00-b3b9-dcef0efacbc9","Type":"ContainerStarted","Data":"e609248f4cf2c4fb2abcca95fdfd3c5f3aaba068fb8e1f2a926b4037839ea5a4"}
Sep 17 17:46:58.232114 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:58.232094    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:46:58.282013 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:58.281910    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f654675b-twgwm" podStartSLOduration=0.931362966 podStartE2EDuration="1.281894705s" totalImagesPullingTime="350.531739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:58.281008638 +0000 UTC m=+793.326013728" watchObservedRunningTime="2026-09-17 17:46:58.281894705 +0000 UTC m=+793.326899794"
Sep 17 17:46:58.305730 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:58.305699    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ph788"]
Sep 17 17:46:58.315359 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:58.315325    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-ph788"]
Sep 17 17:46:59.473264 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:46:59.473230    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f47e26-36cc-4c9b-a0f7-dfe4de280e96" path="/var/lib/kubelet/pods/08f47e26-36cc-4c9b-a0f7-dfe4de280e96/volumes"
Sep 17 17:47:29.241412 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:47:29.241372    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f654675b-twgwm"
Sep 17 17:49:17.906626 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.906585    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"]
Sep 17 17:49:17.907222 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.906943    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f47e26-36cc-4c9b-a0f7-dfe4de280e96" containerName="manager"
Sep 17 17:49:17.907222 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.906960    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f47e26-36cc-4c9b-a0f7-dfe4de280e96" containerName="manager"
Sep 17 17:49:17.907222 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.907031    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f47e26-36cc-4c9b-a0f7-dfe4de280e96" containerName="manager"
Sep 17 17:49:17.910078 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.910057    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:17.912988 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.912968    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:49:17.913082 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.913065    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:49:17.913185 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.913160    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:49:17.913268 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.913192    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q5zmc\""
Sep 17 17:49:17.913268 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.913207    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:49:17.913517 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.913501    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:49:17.913613 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.913508    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:49:17.932107 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:17.932038    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"]
Sep 17 17:49:18.005653 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.005623    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5bpt\" (UniqueName: \"kubernetes.io/projected/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-api-access-j5bpt\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.005813 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.005660    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f45ab63-f2fe-4f0d-a50e-279935e78962-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.005813 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.005802    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.005893 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.005863    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f45ab63-f2fe-4f0d-a50e-279935e78962-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.005932 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.005907    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.106467 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.106436    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bpt\" (UniqueName: \"kubernetes.io/projected/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-api-access-j5bpt\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.106467 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.106471    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f45ab63-f2fe-4f0d-a50e-279935e78962-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.106722 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.106489    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.106722 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.106512    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f45ab63-f2fe-4f0d-a50e-279935e78962-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.106722 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.106544    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.106934 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.106912    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f45ab63-f2fe-4f0d-a50e-279935e78962-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.107174 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.107156    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.107264 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.107247    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.109062 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.109044    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f45ab63-f2fe-4f0d-a50e-279935e78962-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.116749 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.116722    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bpt\" (UniqueName: \"kubernetes.io/projected/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-api-access-j5bpt\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.220939 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.220865    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:18.503350 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.503275    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"]
Sep 17 17:49:18.506741 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:49:18.506716    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f45ab63_f2fe_4f0d_a50e_279935e78962.slice/crio-d734039ef28073ca8dda9248e0ba630cdc5ce0bf21abac0947307230df5ad440 WatchSource:0}: Error finding container d734039ef28073ca8dda9248e0ba630cdc5ce0bf21abac0947307230df5ad440: Status 404 returned error can't find the container with id d734039ef28073ca8dda9248e0ba630cdc5ce0bf21abac0947307230df5ad440
Sep 17 17:49:18.685749 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:18.685715    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" event={"ID":"8f45ab63-f2fe-4f0d-a50e-279935e78962","Type":"ContainerStarted","Data":"d734039ef28073ca8dda9248e0ba630cdc5ce0bf21abac0947307230df5ad440"}
Sep 17 17:49:21.698778 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:21.698689    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" event={"ID":"8f45ab63-f2fe-4f0d-a50e-279935e78962","Type":"ContainerStarted","Data":"b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92"}
Sep 17 17:49:25.712185 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:25.712151    2521 generic.go:356] "Generic (PLEG): container finished" podID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerID="b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92" exitCode=0
Sep 17 17:49:25.712552 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:25.712223    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" event={"ID":"8f45ab63-f2fe-4f0d-a50e-279935e78962","Type":"ContainerDied","Data":"b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92"}
Sep 17 17:49:39.772516 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:39.772482    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" event={"ID":"8f45ab63-f2fe-4f0d-a50e-279935e78962","Type":"ContainerStarted","Data":"2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010"}
Sep 17 17:49:41.781861 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:41.781781    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" event={"ID":"8f45ab63-f2fe-4f0d-a50e-279935e78962","Type":"ContainerStarted","Data":"d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af"}
Sep 17 17:49:41.782190 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:41.781906    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:41.817574 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:41.817503    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podStartSLOduration=2.593587399 podStartE2EDuration="24.817486405s" totalImagesPullingTime="18.543845504s" totalInitContainerRuntime="3.680053502s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:41.81635353 +0000 UTC m=+956.861358620" watchObservedRunningTime="2026-09-17 17:49:41.817486405 +0000 UTC m=+956.862491517"
Sep 17 17:49:42.786173 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:42.786143    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:42.787280 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:42.787249    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 17 17:49:43.789604 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:43.789534    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 17 17:49:48.793662 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:48.793634    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:49:48.794122 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:48.794098    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 17 17:49:58.795018 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:49:58.794973    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 17 17:50:08.795094 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:50:08.795007    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 17 17:50:18.794377 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:50:18.794337    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 17 17:50:28.795048 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:50:28.795008    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 17 17:50:38.794463 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:50:38.794423    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 17 17:50:48.794719 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:50:48.794676    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 17 17:50:58.794716 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:50:58.794681    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:50:59.905796 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:50:59.905763    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"]
Sep 17 17:50:59.906211 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:50:59.905980    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container" containerID="cri-o://2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010" gracePeriod=30
Sep 17 17:50:59.906211 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:50:59.906017    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kube-rbac-proxy" containerID="cri-o://d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af" gracePeriod=30
Sep 17 17:51:00.055640 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:00.055607    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" event={"ID":"8f45ab63-f2fe-4f0d-a50e-279935e78962","Type":"ContainerDied","Data":"d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af"}
Sep 17 17:51:00.055768 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:00.055608    2521 generic.go:356] "Generic (PLEG): container finished" podID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerID="d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af" exitCode=2
Sep 17 17:51:02.878771 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.878738    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"]
Sep 17 17:51:02.881880 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.881863    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:02.884302 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.884271    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-57a06-predictor-serving-cert\""
Sep 17 17:51:02.884432 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.884363    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-57a06-kube-rbac-proxy-sar-config\""
Sep 17 17:51:02.895382 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.895360    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"]
Sep 17 17:51:02.995773 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.995740    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c62233f0-cd55-4b24-b693-28d98417877d-proxy-tls\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:02.995773 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.995782    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwnp\" (UniqueName: \"kubernetes.io/projected/c62233f0-cd55-4b24-b693-28d98417877d-kube-api-access-gxwnp\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:02.996017 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.995869    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62233f0-cd55-4b24-b693-28d98417877d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:02.996017 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.995897    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:02.996017 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:02.995922    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.096621 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.096581    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.096806 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.096672    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c62233f0-cd55-4b24-b693-28d98417877d-proxy-tls\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.096806 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.096704    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwnp\" (UniqueName: \"kubernetes.io/projected/c62233f0-cd55-4b24-b693-28d98417877d-kube-api-access-gxwnp\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.096806 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.096790    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62233f0-cd55-4b24-b693-28d98417877d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.096969 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.096816    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.097377 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.097350    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.097487 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.097352    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62233f0-cd55-4b24-b693-28d98417877d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.097580 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.097542    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.099274 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.099251    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c62233f0-cd55-4b24-b693-28d98417877d-proxy-tls\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.106536 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.106514    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwnp\" (UniqueName: \"kubernetes.io/projected/c62233f0-cd55-4b24-b693-28d98417877d-kube-api-access-gxwnp\") pod \"isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.191919 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.191879    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:03.473724 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.473696    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"]
Sep 17 17:51:03.474875 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:51:03.474851    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62233f0_cd55_4b24_b693_28d98417877d.slice/crio-056e5dca1bca43962c68867798f4fe798efb72cb078ab80db984b20debb2803b WatchSource:0}: Error finding container 056e5dca1bca43962c68867798f4fe798efb72cb078ab80db984b20debb2803b: Status 404 returned error can't find the container with id 056e5dca1bca43962c68867798f4fe798efb72cb078ab80db984b20debb2803b
Sep 17 17:51:03.790869 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:03.790766    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused"
Sep 17 17:51:04.070154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.070073    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" event={"ID":"c62233f0-cd55-4b24-b693-28d98417877d","Type":"ContainerStarted","Data":"88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9"}
Sep 17 17:51:04.070154 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.070132    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" event={"ID":"c62233f0-cd55-4b24-b693-28d98417877d","Type":"ContainerStarted","Data":"056e5dca1bca43962c68867798f4fe798efb72cb078ab80db984b20debb2803b"}
Sep 17 17:51:04.752854 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.752828    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:51:04.809755 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.809683    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5bpt\" (UniqueName: \"kubernetes.io/projected/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-api-access-j5bpt\") pod \"8f45ab63-f2fe-4f0d-a50e-279935e78962\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") "
Sep 17 17:51:04.809755 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.809723    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-rbac-proxy-sar-config\") pod \"8f45ab63-f2fe-4f0d-a50e-279935e78962\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") "
Sep 17 17:51:04.809960 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.809770    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f45ab63-f2fe-4f0d-a50e-279935e78962-kserve-provision-location\") pod \"8f45ab63-f2fe-4f0d-a50e-279935e78962\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") "
Sep 17 17:51:04.809960 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.809801    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f45ab63-f2fe-4f0d-a50e-279935e78962-proxy-tls\") pod \"8f45ab63-f2fe-4f0d-a50e-279935e78962\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") "
Sep 17 17:51:04.809960 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.809841    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-cabundle-cert\") pod \"8f45ab63-f2fe-4f0d-a50e-279935e78962\" (UID: \"8f45ab63-f2fe-4f0d-a50e-279935e78962\") "
Sep 17 17:51:04.810136 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.810108    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45ab63-f2fe-4f0d-a50e-279935e78962-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f45ab63-f2fe-4f0d-a50e-279935e78962" (UID: "8f45ab63-f2fe-4f0d-a50e-279935e78962"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:04.810193 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.810169    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f45ab63-f2fe-4f0d-a50e-279935e78962" (UID: "8f45ab63-f2fe-4f0d-a50e-279935e78962"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:04.810232 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.810209    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f45ab63-f2fe-4f0d-a50e-279935e78962" (UID: "8f45ab63-f2fe-4f0d-a50e-279935e78962"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:04.811935 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.811916    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-api-access-j5bpt" (OuterVolumeSpecName: "kube-api-access-j5bpt") pod "8f45ab63-f2fe-4f0d-a50e-279935e78962" (UID: "8f45ab63-f2fe-4f0d-a50e-279935e78962"). InnerVolumeSpecName "kube-api-access-j5bpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:04.811990 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.811960    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f45ab63-f2fe-4f0d-a50e-279935e78962-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f45ab63-f2fe-4f0d-a50e-279935e78962" (UID: "8f45ab63-f2fe-4f0d-a50e-279935e78962"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:04.910573 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.910534    2521 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f45ab63-f2fe-4f0d-a50e-279935e78962-kserve-provision-location\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:51:04.910728 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.910581    2521 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f45ab63-f2fe-4f0d-a50e-279935e78962-proxy-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:51:04.910728 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.910593    2521 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-cabundle-cert\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:51:04.910728 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.910602    2521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5bpt\" (UniqueName: \"kubernetes.io/projected/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-api-access-j5bpt\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:51:04.910728 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:04.910611    2521 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f45ab63-f2fe-4f0d-a50e-279935e78962-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:51:05.076525 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.076441    2521 generic.go:356] "Generic (PLEG): container finished" podID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerID="2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010" exitCode=0
Sep 17 17:51:05.076525 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.076505    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"
Sep 17 17:51:05.076970 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.076531    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" event={"ID":"8f45ab63-f2fe-4f0d-a50e-279935e78962","Type":"ContainerDied","Data":"2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010"}
Sep 17 17:51:05.076970 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.076616    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4" event={"ID":"8f45ab63-f2fe-4f0d-a50e-279935e78962","Type":"ContainerDied","Data":"d734039ef28073ca8dda9248e0ba630cdc5ce0bf21abac0947307230df5ad440"}
Sep 17 17:51:05.076970 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.076645    2521 scope.go:117] "RemoveContainer" containerID="d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af"
Sep 17 17:51:05.084676 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.084658    2521 scope.go:117] "RemoveContainer" containerID="2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010"
Sep 17 17:51:05.091785 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.091769    2521 scope.go:117] "RemoveContainer" containerID="b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92"
Sep 17 17:51:05.099098 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.098641    2521 scope.go:117] "RemoveContainer" containerID="d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af"
Sep 17 17:51:05.099098 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:51:05.099060    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af\": container with ID starting with d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af not found: ID does not exist" containerID="d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af"
Sep 17 17:51:05.099269 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.099106    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af"} err="failed to get container status \"d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af\": rpc error: code = NotFound desc = could not find container \"d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af\": container with ID starting with d4415cf02ba6ebf458fede04b72eed97dae4af77a30b1137f4b0eb19157c23af not found: ID does not exist"
Sep 17 17:51:05.099269 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.099130    2521 scope.go:117] "RemoveContainer" containerID="2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010"
Sep 17 17:51:05.099427 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:51:05.099406    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010\": container with ID starting with 2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010 not found: ID does not exist" containerID="2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010"
Sep 17 17:51:05.099491 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.099435    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010"} err="failed to get container status \"2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010\": rpc error: code = NotFound desc = could not find container \"2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010\": container with ID starting with 2845d89d8c8055fd87b457cfb3158d11f0aeb77b8061c4fd023a79f7c7e56010 not found: ID does not exist"
Sep 17 17:51:05.099491 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.099454    2521 scope.go:117] "RemoveContainer" containerID="b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92"
Sep 17 17:51:05.100157 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:51:05.100139    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92\": container with ID starting with b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92 not found: ID does not exist" containerID="b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92"
Sep 17 17:51:05.100211 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.100164    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92"} err="failed to get container status \"b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92\": rpc error: code = NotFound desc = could not find container \"b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92\": container with ID starting with b4dd6f6391766320ad3722f529f2f7bec4928fbe970ba0222ef79b6424d0ef92 not found: ID does not exist"
Sep 17 17:51:05.100249 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.100229    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"]
Sep 17 17:51:05.105168 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.105138    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-tpcd4"]
Sep 17 17:51:05.474421 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:05.474388    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" path="/var/lib/kubelet/pods/8f45ab63-f2fe-4f0d-a50e-279935e78962/volumes"
Sep 17 17:51:07.519088 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.519058    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"]
Sep 17 17:51:07.519375 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.519300    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container"
Sep 17 17:51:07.519375 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.519310    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container"
Sep 17 17:51:07.519375 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.519323    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="storage-initializer"
Sep 17 17:51:07.519375 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.519329    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="storage-initializer"
Sep 17 17:51:07.519375 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.519335    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kube-rbac-proxy"
Sep 17 17:51:07.519375 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.519341    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kube-rbac-proxy"
Sep 17 17:51:07.519570 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.519382    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kube-rbac-proxy"
Sep 17 17:51:07.519570 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.519390    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f45ab63-f2fe-4f0d-a50e-279935e78962" containerName="kserve-container"
Sep 17 17:51:07.522225 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.522204    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.526990 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.526964    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lwk\" (UniqueName: \"kubernetes.io/projected/26590278-2cfa-4879-9113-592c71cb00f8-kube-api-access-k4lwk\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.527094 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.526999    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26590278-2cfa-4879-9113-592c71cb00f8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.527094 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.527018    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.527094 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.527048    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26590278-2cfa-4879-9113-592c71cb00f8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.527197 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.527101    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.539973 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.539951    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:51:07.540230 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.540215    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:51:07.550611 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.550587    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"]
Sep 17 17:51:07.627447 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.627416    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lwk\" (UniqueName: \"kubernetes.io/projected/26590278-2cfa-4879-9113-592c71cb00f8-kube-api-access-k4lwk\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.627447 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.627447    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26590278-2cfa-4879-9113-592c71cb00f8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.627683 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.627468    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.627683 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.627495    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26590278-2cfa-4879-9113-592c71cb00f8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.627683 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.627528    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.627935 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.627910    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26590278-2cfa-4879-9113-592c71cb00f8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.628205 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.628181    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.628284 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.628246    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.629940 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.629920    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26590278-2cfa-4879-9113-592c71cb00f8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.639387 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.639367    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lwk\" (UniqueName: \"kubernetes.io/projected/26590278-2cfa-4879-9113-592c71cb00f8-kube-api-access-k4lwk\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:07.849834 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:07.849747    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:08.091646 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:08.091616    2521 generic.go:356] "Generic (PLEG): container finished" podID="c62233f0-cd55-4b24-b693-28d98417877d" containerID="88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9" exitCode=0
Sep 17 17:51:08.091817 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:08.091668    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" event={"ID":"c62233f0-cd55-4b24-b693-28d98417877d","Type":"ContainerDied","Data":"88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9"}
Sep 17 17:51:08.124834 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:08.124806    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"]
Sep 17 17:51:08.127303 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:51:08.127278    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26590278_2cfa_4879_9113_592c71cb00f8.slice/crio-0418d1120ea4204527684888c66f22d397eaf1d8523128b5dd472ff14d9af41a WatchSource:0}: Error finding container 0418d1120ea4204527684888c66f22d397eaf1d8523128b5dd472ff14d9af41a: Status 404 returned error can't find the container with id 0418d1120ea4204527684888c66f22d397eaf1d8523128b5dd472ff14d9af41a
Sep 17 17:51:09.099819 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:09.099515    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" event={"ID":"26590278-2cfa-4879-9113-592c71cb00f8","Type":"ContainerStarted","Data":"67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757"}
Sep 17 17:51:09.099819 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:09.099611    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" event={"ID":"26590278-2cfa-4879-9113-592c71cb00f8","Type":"ContainerStarted","Data":"0418d1120ea4204527684888c66f22d397eaf1d8523128b5dd472ff14d9af41a"}
Sep 17 17:51:13.117857 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:13.117824    2521 generic.go:356] "Generic (PLEG): container finished" podID="26590278-2cfa-4879-9113-592c71cb00f8" containerID="67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757" exitCode=0
Sep 17 17:51:13.118329 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:13.117904    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" event={"ID":"26590278-2cfa-4879-9113-592c71cb00f8","Type":"ContainerDied","Data":"67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757"}
Sep 17 17:51:14.124338 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:14.124302    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" event={"ID":"26590278-2cfa-4879-9113-592c71cb00f8","Type":"ContainerStarted","Data":"109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39"}
Sep 17 17:51:14.124728 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:14.124364    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" event={"ID":"26590278-2cfa-4879-9113-592c71cb00f8","Type":"ContainerStarted","Data":"b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52"}
Sep 17 17:51:14.124728 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:14.124693    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:14.124805 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:14.124754    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:14.126129 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:14.126102    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:51:14.157021 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:14.156971    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podStartSLOduration=3.111439235 podStartE2EDuration="7.156953024s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.045513789s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:14.155529364 +0000 UTC m=+1049.200534468" watchObservedRunningTime="2026-09-17 17:51:14.156953024 +0000 UTC m=+1049.201958112"
Sep 17 17:51:15.129421 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:15.129378    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:51:20.135984 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:20.135736    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:51:20.136406 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:20.136373    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:51:29.187385 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:29.187347    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" event={"ID":"c62233f0-cd55-4b24-b693-28d98417877d","Type":"ContainerStarted","Data":"28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4"}
Sep 17 17:51:29.187804 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:29.187411    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" event={"ID":"c62233f0-cd55-4b24-b693-28d98417877d","Type":"ContainerStarted","Data":"a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e"}
Sep 17 17:51:29.187804 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:29.187688    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:29.187891 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:29.187852    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:29.189108 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:29.189081    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 17:51:29.216216 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:29.216174    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podStartSLOduration=2.868999719 podStartE2EDuration="27.216156703s" totalImagesPullingTime="20.399104005s" totalInitContainerRuntime="3.948052979s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:29.214840515 +0000 UTC m=+1064.259845604" watchObservedRunningTime="2026-09-17 17:51:29.216156703 +0000 UTC m=+1064.261161792"
Sep 17 17:51:30.136672 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:30.136630    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:51:30.191408 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:30.191365    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 17:51:35.196437 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:35.196407    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:51:35.196948 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:35.196920    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 17:51:40.137146 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:40.137064    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:51:45.197060 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:45.197012    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 17:51:50.136793 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:50.136750    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:51:55.197784 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:51:55.197736    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 17:52:00.136419 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:00.136381    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:52:05.197824 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:05.197786    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 17:52:10.136764 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:10.136721    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:52:15.196868 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:15.196831    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 17:52:20.137144 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:20.137100    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:52:25.197471 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:25.197427    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 17:52:30.137728 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:30.137697    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:52:35.197965 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:35.197929    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:52:37.738509 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:37.738472    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"]
Sep 17 17:52:37.738941 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:37.738736    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" containerID="cri-o://b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52" gracePeriod=30
Sep 17 17:52:37.738941 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:37.738784    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kube-rbac-proxy" containerID="cri-o://109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39" gracePeriod=30
Sep 17 17:52:38.429903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:38.429874    2521 generic.go:356] "Generic (PLEG): container finished" podID="26590278-2cfa-4879-9113-592c71cb00f8" containerID="109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39" exitCode=2
Sep 17 17:52:38.429903 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:38.429908    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" event={"ID":"26590278-2cfa-4879-9113-592c71cb00f8","Type":"ContainerDied","Data":"109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39"}
Sep 17 17:52:40.130819 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:40.130769    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 17 17:52:40.136436 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:40.136408    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 17:52:42.376954 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.376929    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:52:42.446104 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.446070    2521 generic.go:356] "Generic (PLEG): container finished" podID="26590278-2cfa-4879-9113-592c71cb00f8" containerID="b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52" exitCode=0
Sep 17 17:52:42.446253 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.446125    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" event={"ID":"26590278-2cfa-4879-9113-592c71cb00f8","Type":"ContainerDied","Data":"b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52"}
Sep 17 17:52:42.446253 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.446153    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"
Sep 17 17:52:42.446253 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.446187    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm" event={"ID":"26590278-2cfa-4879-9113-592c71cb00f8","Type":"ContainerDied","Data":"0418d1120ea4204527684888c66f22d397eaf1d8523128b5dd472ff14d9af41a"}
Sep 17 17:52:42.446253 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.446225    2521 scope.go:117] "RemoveContainer" containerID="109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39"
Sep 17 17:52:42.454160 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.454139    2521 scope.go:117] "RemoveContainer" containerID="b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52"
Sep 17 17:52:42.458310 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.458290    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26590278-2cfa-4879-9113-592c71cb00f8-kserve-provision-location\") pod \"26590278-2cfa-4879-9113-592c71cb00f8\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") "
Sep 17 17:52:42.458405 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.458328    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26590278-2cfa-4879-9113-592c71cb00f8-proxy-tls\") pod \"26590278-2cfa-4879-9113-592c71cb00f8\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") "
Sep 17 17:52:42.458405 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.458357    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4lwk\" (UniqueName: \"kubernetes.io/projected/26590278-2cfa-4879-9113-592c71cb00f8-kube-api-access-k4lwk\") pod \"26590278-2cfa-4879-9113-592c71cb00f8\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") "
Sep 17 17:52:42.458405 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.458380    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-cabundle-cert\") pod \"26590278-2cfa-4879-9113-592c71cb00f8\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") "
Sep 17 17:52:42.458514 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.458419    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-kube-rbac-proxy-sar-config\") pod \"26590278-2cfa-4879-9113-592c71cb00f8\" (UID: \"26590278-2cfa-4879-9113-592c71cb00f8\") "
Sep 17 17:52:42.458693 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.458647    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26590278-2cfa-4879-9113-592c71cb00f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26590278-2cfa-4879-9113-592c71cb00f8" (UID: "26590278-2cfa-4879-9113-592c71cb00f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:42.458873 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.458785    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26590278-2cfa-4879-9113-592c71cb00f8" (UID: "26590278-2cfa-4879-9113-592c71cb00f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:42.459281 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.459230    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26590278-2cfa-4879-9113-592c71cb00f8" (UID: "26590278-2cfa-4879-9113-592c71cb00f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:42.460487 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.460458    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26590278-2cfa-4879-9113-592c71cb00f8-kube-api-access-k4lwk" (OuterVolumeSpecName: "kube-api-access-k4lwk") pod "26590278-2cfa-4879-9113-592c71cb00f8" (UID: "26590278-2cfa-4879-9113-592c71cb00f8"). InnerVolumeSpecName "kube-api-access-k4lwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:42.460837 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.460817    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26590278-2cfa-4879-9113-592c71cb00f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26590278-2cfa-4879-9113-592c71cb00f8" (UID: "26590278-2cfa-4879-9113-592c71cb00f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:42.461887 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.461796    2521 scope.go:117] "RemoveContainer" containerID="67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757"
Sep 17 17:52:42.472808 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.472789    2521 scope.go:117] "RemoveContainer" containerID="109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39"
Sep 17 17:52:42.473105 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:52:42.473085    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39\": container with ID starting with 109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39 not found: ID does not exist" containerID="109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39"
Sep 17 17:52:42.473178 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.473118    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39"} err="failed to get container status \"109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39\": rpc error: code = NotFound desc = could not find container \"109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39\": container with ID starting with 109c928bbfb3eb48b8b97719b65fc58b61e2cb49f0c21ea5eaf03ea218bb0e39 not found: ID does not exist"
Sep 17 17:52:42.473178 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.473145    2521 scope.go:117] "RemoveContainer" containerID="b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52"
Sep 17 17:52:42.473423 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:52:42.473406    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52\": container with ID starting with b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52 not found: ID does not exist" containerID="b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52"
Sep 17 17:52:42.473461 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.473429    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52"} err="failed to get container status \"b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52\": rpc error: code = NotFound desc = could not find container \"b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52\": container with ID starting with b457c5290836e6f1610077e65727c5f50614e1c3e228ff386c5e8402b83e9c52 not found: ID does not exist"
Sep 17 17:52:42.473461 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.473446    2521 scope.go:117] "RemoveContainer" containerID="67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757"
Sep 17 17:52:42.473714 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:52:42.473698    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757\": container with ID starting with 67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757 not found: ID does not exist" containerID="67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757"
Sep 17 17:52:42.473763 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.473723    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757"} err="failed to get container status \"67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757\": rpc error: code = NotFound desc = could not find container \"67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757\": container with ID starting with 67e8072fb97cb22161dc4d8e2b66b8b6286be591ce2f197014489dc430ad8757 not found: ID does not exist"
Sep 17 17:52:42.559501 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.559407    2521 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.559501 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.559446    2521 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26590278-2cfa-4879-9113-592c71cb00f8-kserve-provision-location\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.559501 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.559461    2521 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26590278-2cfa-4879-9113-592c71cb00f8-proxy-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.559501 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.559477    2521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4lwk\" (UniqueName: \"kubernetes.io/projected/26590278-2cfa-4879-9113-592c71cb00f8-kube-api-access-k4lwk\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.559501 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.559495    2521 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26590278-2cfa-4879-9113-592c71cb00f8-cabundle-cert\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:42.782723 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.782691    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"]
Sep 17 17:52:42.789756 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.789722    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-9xxcm"]
Sep 17 17:52:42.805875 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.805847    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"]
Sep 17 17:52:42.806170 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.806157    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="storage-initializer"
Sep 17 17:52:42.806212 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.806172    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="storage-initializer"
Sep 17 17:52:42.806212 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.806182    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container"
Sep 17 17:52:42.806212 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.806189    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container"
Sep 17 17:52:42.806212 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.806202    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kube-rbac-proxy"
Sep 17 17:52:42.806212 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.806207    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kube-rbac-proxy"
Sep 17 17:52:42.806403 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.806247    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kserve-container"
Sep 17 17:52:42.806403 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.806254    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="26590278-2cfa-4879-9113-592c71cb00f8" containerName="kube-rbac-proxy"
Sep 17 17:52:42.810402 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.810347    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:42.812593 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.812552    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-57a06-kube-rbac-proxy-sar-config\""
Sep 17 17:52:42.813387 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.813366    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-57a06-serving-cert\""
Sep 17 17:52:42.823509 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.823488    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"]
Sep 17 17:52:42.861338 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.861311    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/764d00e1-0e90-4935-8204-8e779cf43680-proxy-tls\") pod \"model-chainer-raw-57a06-58698db664-42hgl\" (UID: \"764d00e1-0e90-4935-8204-8e779cf43680\") " pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:42.861487 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.861347    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764d00e1-0e90-4935-8204-8e779cf43680-openshift-service-ca-bundle\") pod \"model-chainer-raw-57a06-58698db664-42hgl\" (UID: \"764d00e1-0e90-4935-8204-8e779cf43680\") " pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:42.952497 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.952466    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"]
Sep 17 17:52:42.955829 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.955808    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:42.958830 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.958810    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 17:52:42.958952 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.958843    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 17:52:42.961889 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.961865    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/764d00e1-0e90-4935-8204-8e779cf43680-proxy-tls\") pod \"model-chainer-raw-57a06-58698db664-42hgl\" (UID: \"764d00e1-0e90-4935-8204-8e779cf43680\") " pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:42.961986 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.961918    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764d00e1-0e90-4935-8204-8e779cf43680-openshift-service-ca-bundle\") pod \"model-chainer-raw-57a06-58698db664-42hgl\" (UID: \"764d00e1-0e90-4935-8204-8e779cf43680\") " pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:42.962686 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.962666    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764d00e1-0e90-4935-8204-8e779cf43680-openshift-service-ca-bundle\") pod \"model-chainer-raw-57a06-58698db664-42hgl\" (UID: \"764d00e1-0e90-4935-8204-8e779cf43680\") " pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:42.964337 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.964314    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/764d00e1-0e90-4935-8204-8e779cf43680-proxy-tls\") pod \"model-chainer-raw-57a06-58698db664-42hgl\" (UID: \"764d00e1-0e90-4935-8204-8e779cf43680\") " pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:42.980304 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:42.980279    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"]
Sep 17 17:52:43.062874 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.062796    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.062874 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.062850    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.062874 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.062868    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86240d29-aa62-40c6-bda4-201019d5a381-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.063096 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.062943    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86240d29-aa62-40c6-bda4-201019d5a381-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.063096 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.062983    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc4td\" (UniqueName: \"kubernetes.io/projected/86240d29-aa62-40c6-bda4-201019d5a381-kube-api-access-jc4td\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.120892 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.120854    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:43.164337 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.164305    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.164490 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.164341    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86240d29-aa62-40c6-bda4-201019d5a381-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.164552 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.164494    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86240d29-aa62-40c6-bda4-201019d5a381-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.164624 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.164545    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc4td\" (UniqueName: \"kubernetes.io/projected/86240d29-aa62-40c6-bda4-201019d5a381-kube-api-access-jc4td\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.164624 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.164600    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.164782 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.164749    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86240d29-aa62-40c6-bda4-201019d5a381-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.165158 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.165134    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.165284 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.165262    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.167704 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.167506    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86240d29-aa62-40c6-bda4-201019d5a381-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.179075 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.179051    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc4td\" (UniqueName: \"kubernetes.io/projected/86240d29-aa62-40c6-bda4-201019d5a381-kube-api-access-jc4td\") pod \"isvc-canary-stop-predictor-5cdcf6545-hppnb\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.265830 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.265801    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:43.396671 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.396641    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"]
Sep 17 17:52:43.399589 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:52:43.399544    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764d00e1_0e90_4935_8204_8e779cf43680.slice/crio-e18e8a4b26fd49ce7a71c9e321e6248169a7e71dad0ef163c1019bd7f6366d7a WatchSource:0}: Error finding container e18e8a4b26fd49ce7a71c9e321e6248169a7e71dad0ef163c1019bd7f6366d7a: Status 404 returned error can't find the container with id e18e8a4b26fd49ce7a71c9e321e6248169a7e71dad0ef163c1019bd7f6366d7a
Sep 17 17:52:43.401641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.401626    2521 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:52:43.451036 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.451007    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" event={"ID":"764d00e1-0e90-4935-8204-8e779cf43680","Type":"ContainerStarted","Data":"e18e8a4b26fd49ce7a71c9e321e6248169a7e71dad0ef163c1019bd7f6366d7a"}
Sep 17 17:52:43.473237 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.473215    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26590278-2cfa-4879-9113-592c71cb00f8" path="/var/lib/kubelet/pods/26590278-2cfa-4879-9113-592c71cb00f8/volumes"
Sep 17 17:52:43.550804 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:43.550785    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"]
Sep 17 17:52:43.553900 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:52:43.553873    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86240d29_aa62_40c6_bda4_201019d5a381.slice/crio-e1a95de0a03a356cf81e1e895b04476fe3a88b54dc83c17b0fbdb18b5899e8fc WatchSource:0}: Error finding container e1a95de0a03a356cf81e1e895b04476fe3a88b54dc83c17b0fbdb18b5899e8fc: Status 404 returned error can't find the container with id e1a95de0a03a356cf81e1e895b04476fe3a88b54dc83c17b0fbdb18b5899e8fc
Sep 17 17:52:44.457633 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:44.457588    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" event={"ID":"86240d29-aa62-40c6-bda4-201019d5a381","Type":"ContainerStarted","Data":"9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a"}
Sep 17 17:52:44.458104 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:44.457664    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" event={"ID":"86240d29-aa62-40c6-bda4-201019d5a381","Type":"ContainerStarted","Data":"e1a95de0a03a356cf81e1e895b04476fe3a88b54dc83c17b0fbdb18b5899e8fc"}
Sep 17 17:52:46.466616 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:46.466581    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" event={"ID":"764d00e1-0e90-4935-8204-8e779cf43680","Type":"ContainerStarted","Data":"30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798"}
Sep 17 17:52:46.467049 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:46.466699    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:46.487284 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:46.486761    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" podStartSLOduration=2.35294005 podStartE2EDuration="4.486744868s" totalImagesPullingTime="2.133804818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:46.486204851 +0000 UTC m=+1141.531209942" watchObservedRunningTime="2026-09-17 17:52:46.486744868 +0000 UTC m=+1141.531749959"
Sep 17 17:52:48.475925 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:48.475892    2521 generic.go:356] "Generic (PLEG): container finished" podID="86240d29-aa62-40c6-bda4-201019d5a381" containerID="9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a" exitCode=0
Sep 17 17:52:48.476265 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:48.475962    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" event={"ID":"86240d29-aa62-40c6-bda4-201019d5a381","Type":"ContainerDied","Data":"9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a"}
Sep 17 17:52:49.481287 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:49.481252    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" event={"ID":"86240d29-aa62-40c6-bda4-201019d5a381","Type":"ContainerStarted","Data":"690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c"}
Sep 17 17:52:49.481714 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:49.481311    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" event={"ID":"86240d29-aa62-40c6-bda4-201019d5a381","Type":"ContainerStarted","Data":"7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7"}
Sep 17 17:52:49.481714 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:49.481583    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:49.481821 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:49.481719    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:49.482988 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:49.482963    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:52:49.506748 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:49.506701    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podStartSLOduration=3.434039742 podStartE2EDuration="7.506685035s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.072645293s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:49.506148698 +0000 UTC m=+1144.551153847" watchObservedRunningTime="2026-09-17 17:52:49.506685035 +0000 UTC m=+1144.551690126"
Sep 17 17:52:50.487359 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:50.487298    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:52:52.475989 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:52.475959    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:52:52.882578 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:52.882485    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"]
Sep 17 17:52:52.882736 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:52.882690    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06" containerID="cri-o://30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798" gracePeriod=30
Sep 17 17:52:53.121397 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:53.121365    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"]
Sep 17 17:52:53.121641 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:53.121615    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" containerID="cri-o://a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e" gracePeriod=30
Sep 17 17:52:53.121753 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:53.121665    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kube-rbac-proxy" containerID="cri-o://28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4" gracePeriod=30
Sep 17 17:52:53.498439 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:53.498353    2521 generic.go:356] "Generic (PLEG): container finished" podID="c62233f0-cd55-4b24-b693-28d98417877d" containerID="28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4" exitCode=2
Sep 17 17:52:53.498439 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:53.498421    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" event={"ID":"c62233f0-cd55-4b24-b693-28d98417877d","Type":"ContainerDied","Data":"28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4"}
Sep 17 17:52:55.192164 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:55.192120    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 17 17:52:55.197610 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:55.197549    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 17:52:55.491971 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:55.491885    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:52:55.492311 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:55.492287    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:52:57.360339 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.360315    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:52:57.474290 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.474259    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c62233f0-cd55-4b24-b693-28d98417877d-proxy-tls\") pod \"c62233f0-cd55-4b24-b693-28d98417877d\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") "
Sep 17 17:52:57.474471 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.474302    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-kube-rbac-proxy-sar-config\") pod \"c62233f0-cd55-4b24-b693-28d98417877d\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") "
Sep 17 17:52:57.474471 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.474363    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62233f0-cd55-4b24-b693-28d98417877d-kserve-provision-location\") pod \"c62233f0-cd55-4b24-b693-28d98417877d\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") "
Sep 17 17:52:57.474471 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.474427    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-cabundle-cert\") pod \"c62233f0-cd55-4b24-b693-28d98417877d\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") "
Sep 17 17:52:57.474471 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.474456    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxwnp\" (UniqueName: \"kubernetes.io/projected/c62233f0-cd55-4b24-b693-28d98417877d-kube-api-access-gxwnp\") pod \"c62233f0-cd55-4b24-b693-28d98417877d\" (UID: \"c62233f0-cd55-4b24-b693-28d98417877d\") "
Sep 17 17:52:57.474750 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.474718    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:57.474808 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.474731    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62233f0-cd55-4b24-b693-28d98417877d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c62233f0-cd55-4b24-b693-28d98417877d" (UID: "c62233f0-cd55-4b24-b693-28d98417877d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:57.474808 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.474762    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c62233f0-cd55-4b24-b693-28d98417877d" (UID: "c62233f0-cd55-4b24-b693-28d98417877d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:57.475316 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.475290    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c62233f0-cd55-4b24-b693-28d98417877d" (UID: "c62233f0-cd55-4b24-b693-28d98417877d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:57.476586 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.476545    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62233f0-cd55-4b24-b693-28d98417877d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c62233f0-cd55-4b24-b693-28d98417877d" (UID: "c62233f0-cd55-4b24-b693-28d98417877d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:57.476660 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.476583    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62233f0-cd55-4b24-b693-28d98417877d-kube-api-access-gxwnp" (OuterVolumeSpecName: "kube-api-access-gxwnp") pod "c62233f0-cd55-4b24-b693-28d98417877d" (UID: "c62233f0-cd55-4b24-b693-28d98417877d"). InnerVolumeSpecName "kube-api-access-gxwnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:57.515439 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.515404    2521 generic.go:356] "Generic (PLEG): container finished" podID="c62233f0-cd55-4b24-b693-28d98417877d" containerID="a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e" exitCode=0
Sep 17 17:52:57.515539 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.515460    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"
Sep 17 17:52:57.515539 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.515471    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" event={"ID":"c62233f0-cd55-4b24-b693-28d98417877d","Type":"ContainerDied","Data":"a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e"}
Sep 17 17:52:57.515539 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.515529    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8" event={"ID":"c62233f0-cd55-4b24-b693-28d98417877d","Type":"ContainerDied","Data":"056e5dca1bca43962c68867798f4fe798efb72cb078ab80db984b20debb2803b"}
Sep 17 17:52:57.515656 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.515591    2521 scope.go:117] "RemoveContainer" containerID="28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4"
Sep 17 17:52:57.523819 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.523776    2521 scope.go:117] "RemoveContainer" containerID="a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e"
Sep 17 17:52:57.532863 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.532849    2521 scope.go:117] "RemoveContainer" containerID="88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9"
Sep 17 17:52:57.539302 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.539281    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"]
Sep 17 17:52:57.540646 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.540612    2521 scope.go:117] "RemoveContainer" containerID="28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4"
Sep 17 17:52:57.541008 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:52:57.540982    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4\": container with ID starting with 28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4 not found: ID does not exist" containerID="28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4"
Sep 17 17:52:57.541068 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.541019    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4"} err="failed to get container status \"28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4\": rpc error: code = NotFound desc = could not find container \"28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4\": container with ID starting with 28ffc5ac3c0251738737a9de862175d9e041d21db9e4092bfe447b1fd2244cd4 not found: ID does not exist"
Sep 17 17:52:57.541068 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.541044    2521 scope.go:117] "RemoveContainer" containerID="a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e"
Sep 17 17:52:57.541335 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:52:57.541315    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e\": container with ID starting with a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e not found: ID does not exist" containerID="a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e"
Sep 17 17:52:57.541403 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.541343    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e"} err="failed to get container status \"a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e\": rpc error: code = NotFound desc = could not find container \"a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e\": container with ID starting with a4feb015a7e775841176fe2d1250e094d121a357eb99d862faebcec103b5b45e not found: ID does not exist"
Sep 17 17:52:57.541403 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.541360    2521 scope.go:117] "RemoveContainer" containerID="88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9"
Sep 17 17:52:57.541531 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.541510    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-57a06-predictor-77b5fc4d8f-q72f8"]
Sep 17 17:52:57.541666 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:52:57.541651    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9\": container with ID starting with 88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9 not found: ID does not exist" containerID="88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9"
Sep 17 17:52:57.541704 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.541671    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9"} err="failed to get container status \"88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9\": rpc error: code = NotFound desc = could not find container \"88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9\": container with ID starting with 88f0358becffae23dc830fe5f02b695c32b58c6c91f8702be4237137f7c0dbb9 not found: ID does not exist"
Sep 17 17:52:57.549225 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:52:57.549203    2521 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62233f0_cd55_4b24_b693_28d98417877d.slice/crio-056e5dca1bca43962c68867798f4fe798efb72cb078ab80db984b20debb2803b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62233f0_cd55_4b24_b693_28d98417877d.slice\": RecentStats: unable to find data in memory cache]"
Sep 17 17:52:57.575242 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.575207    2521 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c62233f0-cd55-4b24-b693-28d98417877d-proxy-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:57.575242 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.575237    2521 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:57.575242 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.575246    2521 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c62233f0-cd55-4b24-b693-28d98417877d-kserve-provision-location\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:57.575483 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.575255    2521 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c62233f0-cd55-4b24-b693-28d98417877d-cabundle-cert\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:57.575483 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:57.575264    2521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxwnp\" (UniqueName: \"kubernetes.io/projected/c62233f0-cd55-4b24-b693-28d98417877d-kube-api-access-gxwnp\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:52:59.473991 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:52:59.473958    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c62233f0-cd55-4b24-b693-28d98417877d" path="/var/lib/kubelet/pods/c62233f0-cd55-4b24-b693-28d98417877d/volumes"
Sep 17 17:53:02.474570 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:02.474516    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:05.493011 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:05.492969    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:53:07.474374 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:07.474334    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:07.474741 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:07.474438    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:53:12.475184 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:12.475139    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:15.492855 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:15.492821    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:53:17.475172 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:17.475128    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:22.474254 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:22.474212    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:23.527827 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.527799    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:53:23.608687 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.608655    2521 generic.go:356] "Generic (PLEG): container finished" podID="764d00e1-0e90-4935-8204-8e779cf43680" containerID="30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798" exitCode=0
Sep 17 17:53:23.608832 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.608704    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"
Sep 17 17:53:23.608832 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.608717    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" event={"ID":"764d00e1-0e90-4935-8204-8e779cf43680","Type":"ContainerDied","Data":"30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798"}
Sep 17 17:53:23.608832 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.608758    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl" event={"ID":"764d00e1-0e90-4935-8204-8e779cf43680","Type":"ContainerDied","Data":"e18e8a4b26fd49ce7a71c9e321e6248169a7e71dad0ef163c1019bd7f6366d7a"}
Sep 17 17:53:23.608832 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.608779    2521 scope.go:117] "RemoveContainer" containerID="30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798"
Sep 17 17:53:23.616372 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.616353    2521 scope.go:117] "RemoveContainer" containerID="30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798"
Sep 17 17:53:23.616664 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:53:23.616646    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798\": container with ID starting with 30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798 not found: ID does not exist" containerID="30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798"
Sep 17 17:53:23.616718 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.616672    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798"} err="failed to get container status \"30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798\": rpc error: code = NotFound desc = could not find container \"30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798\": container with ID starting with 30dc3d3537a112e00e3433b1df933c682c3217093aae2f67822419f00d5c5798 not found: ID does not exist"
Sep 17 17:53:23.671612 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.671514    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/764d00e1-0e90-4935-8204-8e779cf43680-proxy-tls\") pod \"764d00e1-0e90-4935-8204-8e779cf43680\" (UID: \"764d00e1-0e90-4935-8204-8e779cf43680\") "
Sep 17 17:53:23.671612 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.671593    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764d00e1-0e90-4935-8204-8e779cf43680-openshift-service-ca-bundle\") pod \"764d00e1-0e90-4935-8204-8e779cf43680\" (UID: \"764d00e1-0e90-4935-8204-8e779cf43680\") "
Sep 17 17:53:23.671923 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.671900    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764d00e1-0e90-4935-8204-8e779cf43680-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "764d00e1-0e90-4935-8204-8e779cf43680" (UID: "764d00e1-0e90-4935-8204-8e779cf43680"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:23.673487 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.673468    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764d00e1-0e90-4935-8204-8e779cf43680-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "764d00e1-0e90-4935-8204-8e779cf43680" (UID: "764d00e1-0e90-4935-8204-8e779cf43680"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:23.773015 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.772977    2521 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764d00e1-0e90-4935-8204-8e779cf43680-openshift-service-ca-bundle\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:53:23.773015 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.773009    2521 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/764d00e1-0e90-4935-8204-8e779cf43680-proxy-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:53:23.935532 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.935451    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"]
Sep 17 17:53:23.938569 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:23.938535    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-57a06-58698db664-42hgl"]
Sep 17 17:53:25.473948 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:25.473913    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="764d00e1-0e90-4935-8204-8e779cf43680" path="/var/lib/kubelet/pods/764d00e1-0e90-4935-8204-8e779cf43680/volumes"
Sep 17 17:53:25.492535 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:25.492507    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:53:35.492356 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:35.492305    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:53:45.492572 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:45.492519    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:53:55.493298 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:53:55.493258    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 17:54:05.492720 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:05.492687    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:54:17.867858 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:17.867826    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"]
Sep 17 17:54:17.868222 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:17.868077    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container" containerID="cri-o://7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7" gracePeriod=30
Sep 17 17:54:17.868222 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:17.868106    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kube-rbac-proxy" containerID="cri-o://690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c" gracePeriod=30
Sep 17 17:54:18.799611 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:18.799577    2521 generic.go:356] "Generic (PLEG): container finished" podID="86240d29-aa62-40c6-bda4-201019d5a381" containerID="690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c" exitCode=2
Sep 17 17:54:18.799611 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:18.799612    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" event={"ID":"86240d29-aa62-40c6-bda4-201019d5a381","Type":"ContainerDied","Data":"690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c"}
Sep 17 17:54:20.487586 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:20.487535    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 17:54:22.712805 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.712775    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:54:22.805107 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.805022    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc4td\" (UniqueName: \"kubernetes.io/projected/86240d29-aa62-40c6-bda4-201019d5a381-kube-api-access-jc4td\") pod \"86240d29-aa62-40c6-bda4-201019d5a381\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") "
Sep 17 17:54:22.805107 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.805083    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86240d29-aa62-40c6-bda4-201019d5a381-kserve-provision-location\") pod \"86240d29-aa62-40c6-bda4-201019d5a381\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") "
Sep 17 17:54:22.805325 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.805113    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86240d29-aa62-40c6-bda4-201019d5a381-proxy-tls\") pod \"86240d29-aa62-40c6-bda4-201019d5a381\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") "
Sep 17 17:54:22.805325 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.805136    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-kube-rbac-proxy-sar-config\") pod \"86240d29-aa62-40c6-bda4-201019d5a381\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") "
Sep 17 17:54:22.805325 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.805174    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-cabundle-cert\") pod \"86240d29-aa62-40c6-bda4-201019d5a381\" (UID: \"86240d29-aa62-40c6-bda4-201019d5a381\") "
Sep 17 17:54:22.805499 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.805455    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86240d29-aa62-40c6-bda4-201019d5a381-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86240d29-aa62-40c6-bda4-201019d5a381" (UID: "86240d29-aa62-40c6-bda4-201019d5a381"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:22.805637 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.805551    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86240d29-aa62-40c6-bda4-201019d5a381" (UID: "86240d29-aa62-40c6-bda4-201019d5a381"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:22.805637 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.805592    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86240d29-aa62-40c6-bda4-201019d5a381" (UID: "86240d29-aa62-40c6-bda4-201019d5a381"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:22.807179 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.807158    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86240d29-aa62-40c6-bda4-201019d5a381-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86240d29-aa62-40c6-bda4-201019d5a381" (UID: "86240d29-aa62-40c6-bda4-201019d5a381"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:22.807179 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.807167    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86240d29-aa62-40c6-bda4-201019d5a381-kube-api-access-jc4td" (OuterVolumeSpecName: "kube-api-access-jc4td") pod "86240d29-aa62-40c6-bda4-201019d5a381" (UID: "86240d29-aa62-40c6-bda4-201019d5a381"). InnerVolumeSpecName "kube-api-access-jc4td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:22.815591 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.815531    2521 generic.go:356] "Generic (PLEG): container finished" podID="86240d29-aa62-40c6-bda4-201019d5a381" containerID="7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7" exitCode=0
Sep 17 17:54:22.815591 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.815588    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" event={"ID":"86240d29-aa62-40c6-bda4-201019d5a381","Type":"ContainerDied","Data":"7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7"}
Sep 17 17:54:22.815740 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.815618    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"
Sep 17 17:54:22.815740 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.815652    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb" event={"ID":"86240d29-aa62-40c6-bda4-201019d5a381","Type":"ContainerDied","Data":"e1a95de0a03a356cf81e1e895b04476fe3a88b54dc83c17b0fbdb18b5899e8fc"}
Sep 17 17:54:22.815740 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.815689    2521 scope.go:117] "RemoveContainer" containerID="690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c"
Sep 17 17:54:22.823157 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.823138    2521 scope.go:117] "RemoveContainer" containerID="7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7"
Sep 17 17:54:22.830190 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.830173    2521 scope.go:117] "RemoveContainer" containerID="9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a"
Sep 17 17:54:22.836813 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.836796    2521 scope.go:117] "RemoveContainer" containerID="690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c"
Sep 17 17:54:22.837081 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:54:22.837063    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c\": container with ID starting with 690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c not found: ID does not exist" containerID="690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c"
Sep 17 17:54:22.837126 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.837090    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c"} err="failed to get container status \"690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c\": rpc error: code = NotFound desc = could not find container \"690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c\": container with ID starting with 690fed95943c3b862db9ffa0f4aa3a80c18eb267dff9e1b65549f97ece34f56c not found: ID does not exist"
Sep 17 17:54:22.837126 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.837107    2521 scope.go:117] "RemoveContainer" containerID="7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7"
Sep 17 17:54:22.837326 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:54:22.837310    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7\": container with ID starting with 7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7 not found: ID does not exist" containerID="7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7"
Sep 17 17:54:22.837375 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.837333    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7"} err="failed to get container status \"7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7\": rpc error: code = NotFound desc = could not find container \"7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7\": container with ID starting with 7f01ceb3dcba2f46fe961db176fdf305c4a59966fb522441737aea2da8ba31c7 not found: ID does not exist"
Sep 17 17:54:22.837375 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.837347    2521 scope.go:117] "RemoveContainer" containerID="9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a"
Sep 17 17:54:22.837693 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:54:22.837550    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a\": container with ID starting with 9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a not found: ID does not exist" containerID="9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a"
Sep 17 17:54:22.837761 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.837697    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a"} err="failed to get container status \"9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a\": rpc error: code = NotFound desc = could not find container \"9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a\": container with ID starting with 9308cf06443ad448dbdd5eb60f4405e9318ff2e570c3fe333a5d991aae1f1f1a not found: ID does not exist"
Sep 17 17:54:22.840875 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.840855    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"]
Sep 17 17:54:22.847793 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.847768    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-hppnb"]
Sep 17 17:54:22.905669 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.905639    2521 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-cabundle-cert\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:54:22.905669 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.905662    2521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc4td\" (UniqueName: \"kubernetes.io/projected/86240d29-aa62-40c6-bda4-201019d5a381-kube-api-access-jc4td\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:54:22.905669 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.905672    2521 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86240d29-aa62-40c6-bda4-201019d5a381-kserve-provision-location\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:54:22.905878 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.905682    2521 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86240d29-aa62-40c6-bda4-201019d5a381-proxy-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:54:22.905878 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:22.905691    2521 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86240d29-aa62-40c6-bda4-201019d5a381-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:54:23.473533 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:23.473502    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86240d29-aa62-40c6-bda4-201019d5a381" path="/var/lib/kubelet/pods/86240d29-aa62-40c6-bda4-201019d5a381/volumes"
Sep 17 17:54:28.205467 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205431    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"]
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205797    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kube-rbac-proxy"
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205815    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kube-rbac-proxy"
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205867    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container"
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205878    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container"
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205895    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container"
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205904    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container"
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205921    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="storage-initializer"
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205929    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="storage-initializer"
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205944    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kube-rbac-proxy"
Sep 17 17:54:28.205947 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205951    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kube-rbac-proxy"
Sep 17 17:54:28.206241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205963    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="storage-initializer"
Sep 17 17:54:28.206241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205971    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="storage-initializer"
Sep 17 17:54:28.206241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205980    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06"
Sep 17 17:54:28.206241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.205988    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06"
Sep 17 17:54:28.206241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.206055    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="764d00e1-0e90-4935-8204-8e779cf43680" containerName="model-chainer-raw-57a06"
Sep 17 17:54:28.206241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.206066    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kserve-container"
Sep 17 17:54:28.206241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.206076    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kube-rbac-proxy"
Sep 17 17:54:28.206241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.206086    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="86240d29-aa62-40c6-bda4-201019d5a381" containerName="kube-rbac-proxy"
Sep 17 17:54:28.206241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.206095    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="c62233f0-cd55-4b24-b693-28d98417877d" containerName="kserve-container"
Sep 17 17:54:28.210667 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.210644    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.213164 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.213140    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:54:28.213286 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.213145    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a7cd73-kube-rbac-proxy-sar-config\""
Sep 17 17:54:28.213418 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.213403    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:54:28.213418 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.213413    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a7cd73-predictor-serving-cert\""
Sep 17 17:54:28.213534 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.213486    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q5zmc\""
Sep 17 17:54:28.213934 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.213911    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:54:28.214064 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.213943    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:54:28.219314 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.219290    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"]
Sep 17 17:54:28.344196 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.344159    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8b4987-3e6b-4095-8448-7d809e49db51-kserve-provision-location\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.344372 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.344207    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be8b4987-3e6b-4095-8448-7d809e49db51-proxy-tls\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.344372 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.344226    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-cabundle-cert\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.344372 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.344269    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.344372 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.344288    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpw4n\" (UniqueName: \"kubernetes.io/projected/be8b4987-3e6b-4095-8448-7d809e49db51-kube-api-access-dpw4n\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.444630 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.444585    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.444807 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.444635    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpw4n\" (UniqueName: \"kubernetes.io/projected/be8b4987-3e6b-4095-8448-7d809e49db51-kube-api-access-dpw4n\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.444807 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.444693    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8b4987-3e6b-4095-8448-7d809e49db51-kserve-provision-location\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.444807 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.444742    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be8b4987-3e6b-4095-8448-7d809e49db51-proxy-tls\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.444807 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.444771    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-cabundle-cert\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.445144 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.445124    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8b4987-3e6b-4095-8448-7d809e49db51-kserve-provision-location\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.445350 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.445327    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.445413 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.445361    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-cabundle-cert\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.447125 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.447101    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be8b4987-3e6b-4095-8448-7d809e49db51-proxy-tls\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.453898 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.453879    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpw4n\" (UniqueName: \"kubernetes.io/projected/be8b4987-3e6b-4095-8448-7d809e49db51-kube-api-access-dpw4n\") pod \"isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") " pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.521102 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.521034    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:28.792046 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.791907    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"]
Sep 17 17:54:28.794527 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:54:28.794498    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8b4987_3e6b_4095_8448_7d809e49db51.slice/crio-fcf8a1c4edf5b24560ee2aeb0251f210a1f97c08627a109dab4997adb86771d9 WatchSource:0}: Error finding container fcf8a1c4edf5b24560ee2aeb0251f210a1f97c08627a109dab4997adb86771d9: Status 404 returned error can't find the container with id fcf8a1c4edf5b24560ee2aeb0251f210a1f97c08627a109dab4997adb86771d9
Sep 17 17:54:28.838834 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:28.838804    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" event={"ID":"be8b4987-3e6b-4095-8448-7d809e49db51","Type":"ContainerStarted","Data":"fcf8a1c4edf5b24560ee2aeb0251f210a1f97c08627a109dab4997adb86771d9"}
Sep 17 17:54:29.844014 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:29.843983    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" event={"ID":"be8b4987-3e6b-4095-8448-7d809e49db51","Type":"ContainerStarted","Data":"ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7"}
Sep 17 17:54:32.854895 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:32.854810    2521 generic.go:356] "Generic (PLEG): container finished" podID="be8b4987-3e6b-4095-8448-7d809e49db51" containerID="ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7" exitCode=0
Sep 17 17:54:32.854895 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:32.854851    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" event={"ID":"be8b4987-3e6b-4095-8448-7d809e49db51","Type":"ContainerDied","Data":"ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7"}
Sep 17 17:54:33.163399 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.163365    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"]
Sep 17 17:54:33.166446 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.166429    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:33.168436 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.168411    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-68381-kube-rbac-proxy-sar-config\""
Sep 17 17:54:33.168553 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.168411    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-68381-serving-cert\""
Sep 17 17:54:33.177932 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.177909    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"]
Sep 17 17:54:33.282189 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.282156    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe3ad14-d260-47ab-9635-e361512d4e10-proxy-tls\") pod \"model-chainer-raw-hpa-68381-695f48ffc-xdcbm\" (UID: \"abe3ad14-d260-47ab-9635-e361512d4e10\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:33.282349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.282259    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abe3ad14-d260-47ab-9635-e361512d4e10-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-68381-695f48ffc-xdcbm\" (UID: \"abe3ad14-d260-47ab-9635-e361512d4e10\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:33.383509 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.383466    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abe3ad14-d260-47ab-9635-e361512d4e10-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-68381-695f48ffc-xdcbm\" (UID: \"abe3ad14-d260-47ab-9635-e361512d4e10\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:33.383509 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.383515    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe3ad14-d260-47ab-9635-e361512d4e10-proxy-tls\") pod \"model-chainer-raw-hpa-68381-695f48ffc-xdcbm\" (UID: \"abe3ad14-d260-47ab-9635-e361512d4e10\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:33.384194 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.384167    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abe3ad14-d260-47ab-9635-e361512d4e10-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-68381-695f48ffc-xdcbm\" (UID: \"abe3ad14-d260-47ab-9635-e361512d4e10\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:33.385833 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.385813    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe3ad14-d260-47ab-9635-e361512d4e10-proxy-tls\") pod \"model-chainer-raw-hpa-68381-695f48ffc-xdcbm\" (UID: \"abe3ad14-d260-47ab-9635-e361512d4e10\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:33.475739 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.475664    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:33.746163 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.746079    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"]
Sep 17 17:54:33.748891 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:54:33.748864    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe3ad14_d260_47ab_9635_e361512d4e10.slice/crio-93fefdb8771f9f6dccd25498bcda5dd2ba1fa792df458e30fbac3ac5fc8df13b WatchSource:0}: Error finding container 93fefdb8771f9f6dccd25498bcda5dd2ba1fa792df458e30fbac3ac5fc8df13b: Status 404 returned error can't find the container with id 93fefdb8771f9f6dccd25498bcda5dd2ba1fa792df458e30fbac3ac5fc8df13b
Sep 17 17:54:33.860907 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.860871    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" event={"ID":"be8b4987-3e6b-4095-8448-7d809e49db51","Type":"ContainerStarted","Data":"f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd"}
Sep 17 17:54:33.861340 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.860944    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" event={"ID":"be8b4987-3e6b-4095-8448-7d809e49db51","Type":"ContainerStarted","Data":"aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa"}
Sep 17 17:54:33.861340 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.861168    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:33.862417 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.862394    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" event={"ID":"abe3ad14-d260-47ab-9635-e361512d4e10","Type":"ContainerStarted","Data":"725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d"}
Sep 17 17:54:33.862519 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.862430    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" event={"ID":"abe3ad14-d260-47ab-9635-e361512d4e10","Type":"ContainerStarted","Data":"93fefdb8771f9f6dccd25498bcda5dd2ba1fa792df458e30fbac3ac5fc8df13b"}
Sep 17 17:54:33.862519 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.862468    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:33.884694 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.884652    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podStartSLOduration=2.196622671 podStartE2EDuration="5.884639522s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.688016851s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:33.882865654 +0000 UTC m=+1248.927870779" watchObservedRunningTime="2026-09-17 17:54:33.884639522 +0000 UTC m=+1248.929644611"
Sep 17 17:54:33.901225 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:33.901184    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" podStartSLOduration=0.901170792 podStartE2EDuration="901.170792ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:33.900357102 +0000 UTC m=+1248.945362181" watchObservedRunningTime="2026-09-17 17:54:33.901170792 +0000 UTC m=+1248.946175880"
Sep 17 17:54:34.866241 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:34.866202    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:34.867627 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:34.867597    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:54:35.870026 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:35.869987    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:54:39.871359 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:39.871285    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:40.874636 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:40.874605    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:54:40.875268 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:40.875222    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:54:43.222445 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:43.222414    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"]
Sep 17 17:54:43.222841 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:43.222660    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381" containerID="cri-o://725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d" gracePeriod=30
Sep 17 17:54:44.870116 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:44.870069    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:49.869533 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:49.869492    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:50.875913 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:50.875874    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:54:54.869682 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:54.869634    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:54.870118 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:54.869730    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:54:59.870042 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:54:59.870001    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:00.875402 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:00.875359    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:55:04.870290 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:04.870246    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:09.870313 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:09.870266    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:10.876075 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:10.876033    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:55:13.875283 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:13.875256    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:55:13.886992 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:13.886972    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abe3ad14-d260-47ab-9635-e361512d4e10-openshift-service-ca-bundle\") pod \"abe3ad14-d260-47ab-9635-e361512d4e10\" (UID: \"abe3ad14-d260-47ab-9635-e361512d4e10\") "
Sep 17 17:55:13.887088 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:13.887013    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe3ad14-d260-47ab-9635-e361512d4e10-proxy-tls\") pod \"abe3ad14-d260-47ab-9635-e361512d4e10\" (UID: \"abe3ad14-d260-47ab-9635-e361512d4e10\") "
Sep 17 17:55:13.887277 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:13.887255    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe3ad14-d260-47ab-9635-e361512d4e10-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "abe3ad14-d260-47ab-9635-e361512d4e10" (UID: "abe3ad14-d260-47ab-9635-e361512d4e10"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:13.888868 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:13.888845    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe3ad14-d260-47ab-9635-e361512d4e10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abe3ad14-d260-47ab-9635-e361512d4e10" (UID: "abe3ad14-d260-47ab-9635-e361512d4e10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:13.987754 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:13.987714    2521 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abe3ad14-d260-47ab-9635-e361512d4e10-openshift-service-ca-bundle\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:55:13.987754 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:13.987744    2521 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abe3ad14-d260-47ab-9635-e361512d4e10-proxy-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:55:14.001736 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:14.001661    2521 generic.go:356] "Generic (PLEG): container finished" podID="abe3ad14-d260-47ab-9635-e361512d4e10" containerID="725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d" exitCode=137
Sep 17 17:55:14.001736 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:14.001707    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"
Sep 17 17:55:14.001911 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:14.001713    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" event={"ID":"abe3ad14-d260-47ab-9635-e361512d4e10","Type":"ContainerDied","Data":"725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d"}
Sep 17 17:55:14.001911 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:14.001812    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm" event={"ID":"abe3ad14-d260-47ab-9635-e361512d4e10","Type":"ContainerDied","Data":"93fefdb8771f9f6dccd25498bcda5dd2ba1fa792df458e30fbac3ac5fc8df13b"}
Sep 17 17:55:14.001911 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:14.001835    2521 scope.go:117] "RemoveContainer" containerID="725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d"
Sep 17 17:55:14.011028 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:14.011011    2521 scope.go:117] "RemoveContainer" containerID="725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d"
Sep 17 17:55:14.011263 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:55:14.011245    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d\": container with ID starting with 725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d not found: ID does not exist" containerID="725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d"
Sep 17 17:55:14.011323 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:14.011270    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d"} err="failed to get container status \"725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d\": rpc error: code = NotFound desc = could not find container \"725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d\": container with ID starting with 725aec761a3ee24294d9a0e67632323ce75bab1b0b6e718e9342d63b7761a65d not found: ID does not exist"
Sep 17 17:55:14.024269 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:14.024245    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"]
Sep 17 17:55:14.028998 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:14.028975    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-68381-695f48ffc-xdcbm"]
Sep 17 17:55:15.474868 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:15.474840    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" path="/var/lib/kubelet/pods/abe3ad14-d260-47ab-9635-e361512d4e10/volumes"
Sep 17 17:55:20.875866 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:20.875826    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:55:30.875491 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:30.875452    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:55:40.876002 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:40.875958    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 17:55:50.877075 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:55:50.877038    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:56:31.464276 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:31.464238    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"]
Sep 17 17:56:31.464833 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:31.464481    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container" containerID="cri-o://aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa" gracePeriod=30
Sep 17 17:56:31.464833 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:31.464518    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kube-rbac-proxy" containerID="cri-o://f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd" gracePeriod=30
Sep 17 17:56:32.283174 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:32.283141    2521 generic.go:356] "Generic (PLEG): container finished" podID="be8b4987-3e6b-4095-8448-7d809e49db51" containerID="f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd" exitCode=2
Sep 17 17:56:32.283355 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:32.283217    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" event={"ID":"be8b4987-3e6b-4095-8448-7d809e49db51","Type":"ContainerDied","Data":"f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd"}
Sep 17 17:56:35.870644 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:35.870602    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 17:56:36.108910 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.108883    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:56:36.266348 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.266301    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8b4987-3e6b-4095-8448-7d809e49db51-kserve-provision-location\") pod \"be8b4987-3e6b-4095-8448-7d809e49db51\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") "
Sep 17 17:56:36.266584 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.266366    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-kube-rbac-proxy-sar-config\") pod \"be8b4987-3e6b-4095-8448-7d809e49db51\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") "
Sep 17 17:56:36.266584 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.266403    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-cabundle-cert\") pod \"be8b4987-3e6b-4095-8448-7d809e49db51\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") "
Sep 17 17:56:36.266584 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.266437    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be8b4987-3e6b-4095-8448-7d809e49db51-proxy-tls\") pod \"be8b4987-3e6b-4095-8448-7d809e49db51\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") "
Sep 17 17:56:36.266584 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.266469    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpw4n\" (UniqueName: \"kubernetes.io/projected/be8b4987-3e6b-4095-8448-7d809e49db51-kube-api-access-dpw4n\") pod \"be8b4987-3e6b-4095-8448-7d809e49db51\" (UID: \"be8b4987-3e6b-4095-8448-7d809e49db51\") "
Sep 17 17:56:36.266809 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.266777    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be8b4987-3e6b-4095-8448-7d809e49db51" (UID: "be8b4987-3e6b-4095-8448-7d809e49db51"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:36.266858 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.266776    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8b4987-3e6b-4095-8448-7d809e49db51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be8b4987-3e6b-4095-8448-7d809e49db51" (UID: "be8b4987-3e6b-4095-8448-7d809e49db51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:36.266858 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.266835    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be8b4987-3e6b-4095-8448-7d809e49db51" (UID: "be8b4987-3e6b-4095-8448-7d809e49db51"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:36.268543 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.268524    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8b4987-3e6b-4095-8448-7d809e49db51-kube-api-access-dpw4n" (OuterVolumeSpecName: "kube-api-access-dpw4n") pod "be8b4987-3e6b-4095-8448-7d809e49db51" (UID: "be8b4987-3e6b-4095-8448-7d809e49db51"). InnerVolumeSpecName "kube-api-access-dpw4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:36.268658 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.268627    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8b4987-3e6b-4095-8448-7d809e49db51-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be8b4987-3e6b-4095-8448-7d809e49db51" (UID: "be8b4987-3e6b-4095-8448-7d809e49db51"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:36.298996 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.298959    2521 generic.go:356] "Generic (PLEG): container finished" podID="be8b4987-3e6b-4095-8448-7d809e49db51" containerID="aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa" exitCode=0
Sep 17 17:56:36.299117 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.299008    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" event={"ID":"be8b4987-3e6b-4095-8448-7d809e49db51","Type":"ContainerDied","Data":"aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa"}
Sep 17 17:56:36.299117 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.299019    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"
Sep 17 17:56:36.299117 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.299056    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd" event={"ID":"be8b4987-3e6b-4095-8448-7d809e49db51","Type":"ContainerDied","Data":"fcf8a1c4edf5b24560ee2aeb0251f210a1f97c08627a109dab4997adb86771d9"}
Sep 17 17:56:36.299117 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.299081    2521 scope.go:117] "RemoveContainer" containerID="f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd"
Sep 17 17:56:36.307551 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.307458    2521 scope.go:117] "RemoveContainer" containerID="aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa"
Sep 17 17:56:36.314291 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.314273    2521 scope.go:117] "RemoveContainer" containerID="ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7"
Sep 17 17:56:36.321096 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.321078    2521 scope.go:117] "RemoveContainer" containerID="f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd"
Sep 17 17:56:36.321343 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:56:36.321325    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd\": container with ID starting with f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd not found: ID does not exist" containerID="f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd"
Sep 17 17:56:36.321385 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.321352    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd"} err="failed to get container status \"f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd\": rpc error: code = NotFound desc = could not find container \"f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd\": container with ID starting with f2582e1782b386c6fd21edc9c5415abf42cfe0e9f8d3f2fe8c975cd4936cbdfd not found: ID does not exist"
Sep 17 17:56:36.321385 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.321368    2521 scope.go:117] "RemoveContainer" containerID="aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa"
Sep 17 17:56:36.321633 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:56:36.321613    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa\": container with ID starting with aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa not found: ID does not exist" containerID="aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa"
Sep 17 17:56:36.321676 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.321637    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa"} err="failed to get container status \"aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa\": rpc error: code = NotFound desc = could not find container \"aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa\": container with ID starting with aca1e2088c140fb0d709be074cbdcdf31bef34b6179e27ebfa32503e707125fa not found: ID does not exist"
Sep 17 17:56:36.321676 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.321649    2521 scope.go:117] "RemoveContainer" containerID="ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7"
Sep 17 17:56:36.321855 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:56:36.321840    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7\": container with ID starting with ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7 not found: ID does not exist" containerID="ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7"
Sep 17 17:56:36.321894 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.321859    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7"} err="failed to get container status \"ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7\": rpc error: code = NotFound desc = could not find container \"ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7\": container with ID starting with ac6e377bf1e77438ccd1ef2cc2712c4abcb50ecf40901bbce5c52a49f73b46e7 not found: ID does not exist"
Sep 17 17:56:36.338953 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.338924    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"]
Sep 17 17:56:36.342866 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.342847    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a7cd73-predictor-6887dfdd9-8q2xd"]
Sep 17 17:56:36.367157 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.367139    2521 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be8b4987-3e6b-4095-8448-7d809e49db51-proxy-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:56:36.367227 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.367158    2521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpw4n\" (UniqueName: \"kubernetes.io/projected/be8b4987-3e6b-4095-8448-7d809e49db51-kube-api-access-dpw4n\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:56:36.367227 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.367168    2521 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8b4987-3e6b-4095-8448-7d809e49db51-kserve-provision-location\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:56:36.367227 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.367177    2521 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:56:36.367227 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:36.367186    2521 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8b4987-3e6b-4095-8448-7d809e49db51-cabundle-cert\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:56:37.474870 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:37.474829    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" path="/var/lib/kubelet/pods/be8b4987-3e6b-4095-8448-7d809e49db51/volumes"
Sep 17 17:56:39.019000 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.018964    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"]
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019187    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019198    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019212    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="storage-initializer"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019218    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="storage-initializer"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019231    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kube-rbac-proxy"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019236    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kube-rbac-proxy"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019243    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019248    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019289    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kube-rbac-proxy"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019298    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe3ad14-d260-47ab-9635-e361512d4e10" containerName="model-chainer-raw-hpa-68381"
Sep 17 17:56:39.019349 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.019305    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8b4987-3e6b-4095-8448-7d809e49db51" containerName="kserve-container"
Sep 17 17:56:39.023469 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.023449    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.026091 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.026065    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-603e1-kube-rbac-proxy-sar-config\""
Sep 17 17:56:39.026091 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.026089    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q5zmc\""
Sep 17 17:56:39.026277 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.026069    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-603e1-predictor-serving-cert\""
Sep 17 17:56:39.026277 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.026070    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:56:39.026465 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.026450    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:56:39.026538 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.026483    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:56:39.026888 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.026873    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:56:39.041247 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.041226    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"]
Sep 17 17:56:39.188946 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.188906    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwgzp\" (UniqueName: \"kubernetes.io/projected/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-api-access-hwgzp\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.188946 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.188948    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.189155 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.188981    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-cabundle-cert\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.189155 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.189007    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.189155 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.189026    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-proxy-tls\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.290025 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.289931    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwgzp\" (UniqueName: \"kubernetes.io/projected/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-api-access-hwgzp\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.290025 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.289972    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.290025 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.289998    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-cabundle-cert\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.290025 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.290019    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.290322 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.290042    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-proxy-tls\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.290433 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.290409    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.290793 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.290768    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.290917 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.290769    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-cabundle-cert\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.292605 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.292584    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-proxy-tls\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.300334 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.300304    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwgzp\" (UniqueName: \"kubernetes.io/projected/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-api-access-hwgzp\") pod \"isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.333607 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.333572    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:39.617001 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:39.616923    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"]
Sep 17 17:56:39.620429 ip-10-0-135-179 kubenswrapper[2521]: W0917 17:56:39.620398    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0a0956_a30e_41f0_9ff1_42db5ddf7540.slice/crio-9442b1fd545d5a7bd928f60a4b9858c56434589ce2d7e6343186b37480f9fca7 WatchSource:0}: Error finding container 9442b1fd545d5a7bd928f60a4b9858c56434589ce2d7e6343186b37480f9fca7: Status 404 returned error can't find the container with id 9442b1fd545d5a7bd928f60a4b9858c56434589ce2d7e6343186b37480f9fca7
Sep 17 17:56:40.317257 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:40.317223    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" event={"ID":"ea0a0956-a30e-41f0-9ff1-42db5ddf7540","Type":"ContainerStarted","Data":"6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021"}
Sep 17 17:56:40.317683 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:40.317291    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" event={"ID":"ea0a0956-a30e-41f0-9ff1-42db5ddf7540","Type":"ContainerStarted","Data":"9442b1fd545d5a7bd928f60a4b9858c56434589ce2d7e6343186b37480f9fca7"}
Sep 17 17:56:44.331423 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:44.331390    2521 generic.go:356] "Generic (PLEG): container finished" podID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerID="6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021" exitCode=0
Sep 17 17:56:44.331813 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:44.331438    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" event={"ID":"ea0a0956-a30e-41f0-9ff1-42db5ddf7540","Type":"ContainerDied","Data":"6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021"}
Sep 17 17:56:45.337101 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:45.337068    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" event={"ID":"ea0a0956-a30e-41f0-9ff1-42db5ddf7540","Type":"ContainerStarted","Data":"c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7"}
Sep 17 17:56:45.337453 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:45.337133    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" event={"ID":"ea0a0956-a30e-41f0-9ff1-42db5ddf7540","Type":"ContainerStarted","Data":"bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f"}
Sep 17 17:56:45.337453 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:45.337330    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:45.361579 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:45.361512    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podStartSLOduration=3.605203025 podStartE2EDuration="7.361497015s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.75629399s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:45.360385106 +0000 UTC m=+1380.405390196" watchObservedRunningTime="2026-09-17 17:56:45.361497015 +0000 UTC m=+1380.406502104"
Sep 17 17:56:46.340418 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:46.340387    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:46.341548 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:46.341524    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:56:47.343628 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:47.343587    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:56:52.348210 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:52.348178    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:56:52.348679 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:56:52.348652    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:57:02.348934 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:57:02.348894    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:57:12.349351 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:57:12.349305    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:57:22.348789 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:57:22.348749    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:57:32.349128 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:57:32.349085    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:57:42.348914 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:57:42.348872    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:57:52.348928 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:57:52.348882    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:58:02.348826 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:58:02.348781    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:58:12.349134 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:58:12.349091    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:58:21.469367 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:58:21.469328    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:58:31.470220 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:58:31.470169    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:58:41.469692 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:58:41.469651    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:58:51.469385 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:58:51.469340    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:59:01.470233 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:01.470185    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:59:11.469701 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:11.469640    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:59:21.473164 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:21.473126    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:59:28.931852 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:28.931818    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"]
Sep 17 17:59:28.932397 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:28.932115    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" containerID="cri-o://bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f" gracePeriod=30
Sep 17 17:59:28.932397 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:28.932160    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kube-rbac-proxy" containerID="cri-o://c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7" gracePeriod=30
Sep 17 17:59:29.897870 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:29.897835    2521 generic.go:356] "Generic (PLEG): container finished" podID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerID="c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7" exitCode=2
Sep 17 17:59:29.897870 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:29.897871    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" event={"ID":"ea0a0956-a30e-41f0-9ff1-42db5ddf7540","Type":"ContainerDied","Data":"c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7"}
Sep 17 17:59:31.469370 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:31.469323    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:59:32.344343 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:32.344295    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 17 17:59:37.344511 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:37.344467    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 17 17:59:38.875533 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.875509    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:59:38.927381 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.927354    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwgzp\" (UniqueName: \"kubernetes.io/projected/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-api-access-hwgzp\") pod \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") "
Sep 17 17:59:38.927530 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.927421    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kserve-provision-location\") pod \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") "
Sep 17 17:59:38.927530 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.927442    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-cabundle-cert\") pod \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") "
Sep 17 17:59:38.927530 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.927457    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-rbac-proxy-sar-config\") pod \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") "
Sep 17 17:59:38.927530 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.927482    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-proxy-tls\") pod \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\" (UID: \"ea0a0956-a30e-41f0-9ff1-42db5ddf7540\") "
Sep 17 17:59:38.927785 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.927747    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea0a0956-a30e-41f0-9ff1-42db5ddf7540" (UID: "ea0a0956-a30e-41f0-9ff1-42db5ddf7540"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:38.927917 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.927891    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea0a0956-a30e-41f0-9ff1-42db5ddf7540" (UID: "ea0a0956-a30e-41f0-9ff1-42db5ddf7540"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:38.927917 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.927899    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea0a0956-a30e-41f0-9ff1-42db5ddf7540" (UID: "ea0a0956-a30e-41f0-9ff1-42db5ddf7540"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:38.929796 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.929771    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea0a0956-a30e-41f0-9ff1-42db5ddf7540" (UID: "ea0a0956-a30e-41f0-9ff1-42db5ddf7540"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:38.929796 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.929779    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-api-access-hwgzp" (OuterVolumeSpecName: "kube-api-access-hwgzp") pod "ea0a0956-a30e-41f0-9ff1-42db5ddf7540" (UID: "ea0a0956-a30e-41f0-9ff1-42db5ddf7540"). InnerVolumeSpecName "kube-api-access-hwgzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:38.930529 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.930503    2521 generic.go:356] "Generic (PLEG): container finished" podID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerID="bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f" exitCode=0
Sep 17 17:59:38.930640 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.930579    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"
Sep 17 17:59:38.930701 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.930669    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" event={"ID":"ea0a0956-a30e-41f0-9ff1-42db5ddf7540","Type":"ContainerDied","Data":"bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f"}
Sep 17 17:59:38.930756 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.930739    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb" event={"ID":"ea0a0956-a30e-41f0-9ff1-42db5ddf7540","Type":"ContainerDied","Data":"9442b1fd545d5a7bd928f60a4b9858c56434589ce2d7e6343186b37480f9fca7"}
Sep 17 17:59:38.930808 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.930782    2521 scope.go:117] "RemoveContainer" containerID="c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7"
Sep 17 17:59:38.941891 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.941875    2521 scope.go:117] "RemoveContainer" containerID="bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f"
Sep 17 17:59:38.949257 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.949242    2521 scope.go:117] "RemoveContainer" containerID="6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021"
Sep 17 17:59:38.955033 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.955010    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"]
Sep 17 17:59:38.956148 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.956126    2521 scope.go:117] "RemoveContainer" containerID="c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7"
Sep 17 17:59:38.956412 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:59:38.956394    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7\": container with ID starting with c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7 not found: ID does not exist" containerID="c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7"
Sep 17 17:59:38.956484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.956424    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7"} err="failed to get container status \"c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7\": rpc error: code = NotFound desc = could not find container \"c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7\": container with ID starting with c75091ffbfa11f07925ef4b8c6ea6bd214271203773a0f4549b6632ef244cfc7 not found: ID does not exist"
Sep 17 17:59:38.956484 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.956447    2521 scope.go:117] "RemoveContainer" containerID="bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f"
Sep 17 17:59:38.956719 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:59:38.956699    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f\": container with ID starting with bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f not found: ID does not exist" containerID="bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f"
Sep 17 17:59:38.956787 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.956728    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f"} err="failed to get container status \"bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f\": rpc error: code = NotFound desc = could not find container \"bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f\": container with ID starting with bbeb3453229be0635193fd087d63e511c59b3b0c106b07300c13fda3ba270f5f not found: ID does not exist"
Sep 17 17:59:38.956787 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.956750    2521 scope.go:117] "RemoveContainer" containerID="6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021"
Sep 17 17:59:38.956978 ip-10-0-135-179 kubenswrapper[2521]: E0917 17:59:38.956961    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021\": container with ID starting with 6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021 not found: ID does not exist" containerID="6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021"
Sep 17 17:59:38.957021 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.956985    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021"} err="failed to get container status \"6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021\": rpc error: code = NotFound desc = could not find container \"6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021\": container with ID starting with 6f017b9a12dde510b258f234495d54222b48dd9950898e93ad89912915aeb021 not found: ID does not exist"
Sep 17 17:59:38.962878 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:38.962859    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-603e1-predictor-5c8b689d74-p9fbb"]
Sep 17 17:59:39.028174 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:39.028112    2521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwgzp\" (UniqueName: \"kubernetes.io/projected/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-api-access-hwgzp\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.028174 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:39.028137    2521 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kserve-provision-location\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.028174 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:39.028152    2521 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-cabundle-cert\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.028174 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:39.028164    2521 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.028174 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:39.028176    2521 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea0a0956-a30e-41f0-9ff1-42db5ddf7540-proxy-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.473801 ip-10-0-135-179 kubenswrapper[2521]: I0917 17:59:39.473768    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" path="/var/lib/kubelet/pods/ea0a0956-a30e-41f0-9ff1-42db5ddf7540/volumes"
Sep 17 18:01:09.525843 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.525806    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"]
Sep 17 18:01:09.526308 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.526054    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kube-rbac-proxy"
Sep 17 18:01:09.526308 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.526067    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kube-rbac-proxy"
Sep 17 18:01:09.526308 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.526076    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container"
Sep 17 18:01:09.526308 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.526081    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container"
Sep 17 18:01:09.526308 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.526093    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="storage-initializer"
Sep 17 18:01:09.526308 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.526099    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="storage-initializer"
Sep 17 18:01:09.526308 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.526152    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kserve-container"
Sep 17 18:01:09.526308 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.526159    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea0a0956-a30e-41f0-9ff1-42db5ddf7540" containerName="kube-rbac-proxy"
Sep 17 18:01:09.529056 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.529039    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.531348 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.531323    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:01:09.531484 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.531347    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:01:09.531628 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.531612    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:01:09.531724 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.531649    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:01:09.531786 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.531722    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q5zmc\""
Sep 17 18:01:09.532289 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.532269    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:01:09.532370 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.532352    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:01:09.540620 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.540600    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"]
Sep 17 18:01:09.554179 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.554156    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.554315 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.554195    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104a153c-1705-44a7-bacb-3eee5c60bc8d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.554315 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.554231    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.554420 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.554313    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104a153c-1705-44a7-bacb-3eee5c60bc8d-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.554420 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.554356    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6j2l\" (UniqueName: \"kubernetes.io/projected/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-api-access-t6j2l\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.655568 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.655533    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.655731 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.655613    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104a153c-1705-44a7-bacb-3eee5c60bc8d-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.655731 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.655648    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6j2l\" (UniqueName: \"kubernetes.io/projected/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-api-access-t6j2l\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.655731 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.655699    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.655895 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.655745    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104a153c-1705-44a7-bacb-3eee5c60bc8d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.656149 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.656114    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104a153c-1705-44a7-bacb-3eee5c60bc8d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.656301 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.656284    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.656400 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.656381    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.658113 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.658092    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104a153c-1705-44a7-bacb-3eee5c60bc8d-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.664259 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.664233    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6j2l\" (UniqueName: \"kubernetes.io/projected/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-api-access-t6j2l\") pod \"isvc-canary-promote-predictor-7cf78f884f-fxk4m\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:09.839391 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:09.839298    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:10.119060 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:10.118990    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"]
Sep 17 18:01:10.121782 ip-10-0-135-179 kubenswrapper[2521]: W0917 18:01:10.121752    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104a153c_1705_44a7_bacb_3eee5c60bc8d.slice/crio-89624f4c74ee3ce09a914c3061d85465420309bf581b2915eaec581c4ce79781 WatchSource:0}: Error finding container 89624f4c74ee3ce09a914c3061d85465420309bf581b2915eaec581c4ce79781: Status 404 returned error can't find the container with id 89624f4c74ee3ce09a914c3061d85465420309bf581b2915eaec581c4ce79781
Sep 17 18:01:10.231725 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:10.231692    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" event={"ID":"104a153c-1705-44a7-bacb-3eee5c60bc8d","Type":"ContainerStarted","Data":"322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e"}
Sep 17 18:01:10.231891 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:10.231758    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" event={"ID":"104a153c-1705-44a7-bacb-3eee5c60bc8d","Type":"ContainerStarted","Data":"89624f4c74ee3ce09a914c3061d85465420309bf581b2915eaec581c4ce79781"}
Sep 17 18:01:14.248661 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:14.248622    2521 generic.go:356] "Generic (PLEG): container finished" podID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerID="322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e" exitCode=0
Sep 17 18:01:14.249036 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:14.248687    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" event={"ID":"104a153c-1705-44a7-bacb-3eee5c60bc8d","Type":"ContainerDied","Data":"322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e"}
Sep 17 18:01:15.254231 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:15.254199    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" event={"ID":"104a153c-1705-44a7-bacb-3eee5c60bc8d","Type":"ContainerStarted","Data":"b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179"}
Sep 17 18:01:15.254686 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:15.254257    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" event={"ID":"104a153c-1705-44a7-bacb-3eee5c60bc8d","Type":"ContainerStarted","Data":"30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a"}
Sep 17 18:01:15.254686 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:15.254497    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:15.277853 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:15.277807    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podStartSLOduration=2.442322867 podStartE2EDuration="6.277791492s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835468625s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:15.276454844 +0000 UTC m=+1650.321459935" watchObservedRunningTime="2026-09-17 18:01:15.277791492 +0000 UTC m=+1650.322796581"
Sep 17 18:01:16.257244 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:16.257214    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:16.258668 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:16.258640    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:01:17.260362 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:17.260324    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:01:22.265238 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:22.265207    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:01:22.265742 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:22.265711    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:01:32.265700 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:32.265652    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:01:42.266388 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:42.266345    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:01:52.266389 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:01:52.266349    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:02:02.265727 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:02.265687    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:02:12.266644 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:12.266602    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:02:22.265721 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:22.265676    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:02:32.267042 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:32.267012    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:02:39.571177 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:39.571147    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"]
Sep 17 18:02:39.571665 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:39.571386    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" containerID="cri-o://30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a" gracePeriod=30
Sep 17 18:02:39.571665 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:39.571432    2521 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kube-rbac-proxy" containerID="cri-o://b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179" gracePeriod=30
Sep 17 18:02:40.542700 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:40.542670    2521 generic.go:356] "Generic (PLEG): container finished" podID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerID="b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179" exitCode=2
Sep 17 18:02:40.542873 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:40.542752    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" event={"ID":"104a153c-1705-44a7-bacb-3eee5c60bc8d","Type":"ContainerDied","Data":"b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179"}
Sep 17 18:02:42.261490 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:42.261431    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 18:02:42.265798 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:42.265771    2521 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 18:02:44.307192 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.307167    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:02:44.336476 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.336451    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6j2l\" (UniqueName: \"kubernetes.io/projected/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-api-access-t6j2l\") pod \"104a153c-1705-44a7-bacb-3eee5c60bc8d\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") "
Sep 17 18:02:44.336629 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.336503    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104a153c-1705-44a7-bacb-3eee5c60bc8d-proxy-tls\") pod \"104a153c-1705-44a7-bacb-3eee5c60bc8d\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") "
Sep 17 18:02:44.336629 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.336539    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104a153c-1705-44a7-bacb-3eee5c60bc8d-kserve-provision-location\") pod \"104a153c-1705-44a7-bacb-3eee5c60bc8d\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") "
Sep 17 18:02:44.336629 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.336594    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-cabundle-cert\") pod \"104a153c-1705-44a7-bacb-3eee5c60bc8d\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") "
Sep 17 18:02:44.336786 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.336631    2521 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-rbac-proxy-sar-config\") pod \"104a153c-1705-44a7-bacb-3eee5c60bc8d\" (UID: \"104a153c-1705-44a7-bacb-3eee5c60bc8d\") "
Sep 17 18:02:44.336961 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.336927    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/104a153c-1705-44a7-bacb-3eee5c60bc8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "104a153c-1705-44a7-bacb-3eee5c60bc8d" (UID: "104a153c-1705-44a7-bacb-3eee5c60bc8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:44.337081 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.336963    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "104a153c-1705-44a7-bacb-3eee5c60bc8d" (UID: "104a153c-1705-44a7-bacb-3eee5c60bc8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:44.337081 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.337051    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "104a153c-1705-44a7-bacb-3eee5c60bc8d" (UID: "104a153c-1705-44a7-bacb-3eee5c60bc8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:44.338631 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.338600    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104a153c-1705-44a7-bacb-3eee5c60bc8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "104a153c-1705-44a7-bacb-3eee5c60bc8d" (UID: "104a153c-1705-44a7-bacb-3eee5c60bc8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:44.338722 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.338636    2521 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-api-access-t6j2l" (OuterVolumeSpecName: "kube-api-access-t6j2l") pod "104a153c-1705-44a7-bacb-3eee5c60bc8d" (UID: "104a153c-1705-44a7-bacb-3eee5c60bc8d"). InnerVolumeSpecName "kube-api-access-t6j2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:44.437933 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.437899    2521 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/104a153c-1705-44a7-bacb-3eee5c60bc8d-proxy-tls\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.437933 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.437928    2521 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/104a153c-1705-44a7-bacb-3eee5c60bc8d-kserve-provision-location\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.437933 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.437938    2521 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-cabundle-cert\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.438166 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.437947    2521 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.438166 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.437955    2521 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6j2l\" (UniqueName: \"kubernetes.io/projected/104a153c-1705-44a7-bacb-3eee5c60bc8d-kube-api-access-t6j2l\") on node \"ip-10-0-135-179.ec2.internal\" DevicePath \"\""
Sep 17 18:02:44.559441 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.559405    2521 generic.go:356] "Generic (PLEG): container finished" podID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerID="30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a" exitCode=0
Sep 17 18:02:44.559609 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.559446    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" event={"ID":"104a153c-1705-44a7-bacb-3eee5c60bc8d","Type":"ContainerDied","Data":"30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a"}
Sep 17 18:02:44.559609 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.559470    2521 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"
Sep 17 18:02:44.559609 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.559505    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m" event={"ID":"104a153c-1705-44a7-bacb-3eee5c60bc8d","Type":"ContainerDied","Data":"89624f4c74ee3ce09a914c3061d85465420309bf581b2915eaec581c4ce79781"}
Sep 17 18:02:44.559609 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.559544    2521 scope.go:117] "RemoveContainer" containerID="b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179"
Sep 17 18:02:44.570020 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.570001    2521 scope.go:117] "RemoveContainer" containerID="30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a"
Sep 17 18:02:44.576992 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.576974    2521 scope.go:117] "RemoveContainer" containerID="322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e"
Sep 17 18:02:44.582488 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.582469    2521 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"]
Sep 17 18:02:44.584098 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.584076    2521 scope.go:117] "RemoveContainer" containerID="b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179"
Sep 17 18:02:44.584347 ip-10-0-135-179 kubenswrapper[2521]: E0917 18:02:44.584327    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179\": container with ID starting with b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179 not found: ID does not exist" containerID="b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179"
Sep 17 18:02:44.584449 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.584358    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179"} err="failed to get container status \"b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179\": rpc error: code = NotFound desc = could not find container \"b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179\": container with ID starting with b47b5e9caa57d30e2eefb6b941f9d9495950b56e603b4199ded9a8039ad63179 not found: ID does not exist"
Sep 17 18:02:44.584449 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.584383    2521 scope.go:117] "RemoveContainer" containerID="30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a"
Sep 17 18:02:44.584665 ip-10-0-135-179 kubenswrapper[2521]: E0917 18:02:44.584647    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a\": container with ID starting with 30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a not found: ID does not exist" containerID="30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a"
Sep 17 18:02:44.584710 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.584677    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a"} err="failed to get container status \"30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a\": rpc error: code = NotFound desc = could not find container \"30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a\": container with ID starting with 30a0c44e9528716a9151e8923a725f52b66cbefcbeb04d37ca515da6040ab99a not found: ID does not exist"
Sep 17 18:02:44.584710 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.584693    2521 scope.go:117] "RemoveContainer" containerID="322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e"
Sep 17 18:02:44.584919 ip-10-0-135-179 kubenswrapper[2521]: E0917 18:02:44.584901    2521 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e\": container with ID starting with 322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e not found: ID does not exist" containerID="322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e"
Sep 17 18:02:44.584960 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.584926    2521 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e"} err="failed to get container status \"322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e\": rpc error: code = NotFound desc = could not find container \"322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e\": container with ID starting with 322a0c27b222253352c5b6955a3831195961d4e6ef1d8b26e596849806dbd34e not found: ID does not exist"
Sep 17 18:02:44.589402 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:44.589373    2521 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-fxk4m"]
Sep 17 18:02:45.474134 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:02:45.474098    2521 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" path="/var/lib/kubelet/pods/104a153c-1705-44a7-bacb-3eee5c60bc8d/volumes"
Sep 17 18:06:54.928848 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:06:54.928773    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vzqnh_2da9dc7f-e3f7-442b-bf4e-badb16e819bd/global-pull-secret-syncer/0.log"
Sep 17 18:06:55.020724 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:06:55.020697    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nx4pl_3c431506-7b50-4923-931f-c07f448fa01e/konnectivity-agent/0.log"
Sep 17 18:06:55.121127 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:06:55.121094    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-179.ec2.internal_75b092477877a2d07a79ef4fdc413311/haproxy/0.log"
Sep 17 18:06:59.568906 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:06:59.568876    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8drvh_7821b4dd-48eb-4488-9761-b3b27be661f4/node-exporter/0.log"
Sep 17 18:06:59.596215 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:06:59.596188    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8drvh_7821b4dd-48eb-4488-9761-b3b27be661f4/kube-rbac-proxy/0.log"
Sep 17 18:06:59.624541 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:06:59.624519    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8drvh_7821b4dd-48eb-4488-9761-b3b27be661f4/init-textfile/0.log"
Sep 17 18:07:02.984250 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.984217    2521 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"]
Sep 17 18:07:02.984662 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.984465    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kube-rbac-proxy"
Sep 17 18:07:02.984662 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.984478    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kube-rbac-proxy"
Sep 17 18:07:02.984662 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.984493    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="storage-initializer"
Sep 17 18:07:02.984662 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.984503    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="storage-initializer"
Sep 17 18:07:02.984662 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.984522    2521 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container"
Sep 17 18:07:02.984662 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.984527    2521 state_mem.go:107] "Deleted CPUSet assignment" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container"
Sep 17 18:07:02.984662 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.984583    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kserve-container"
Sep 17 18:07:02.984662 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.984594    2521 memory_manager.go:368] "RemoveStaleState removing state" podUID="104a153c-1705-44a7-bacb-3eee5c60bc8d" containerName="kube-rbac-proxy"
Sep 17 18:07:02.987361 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.987344    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:02.989684 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.989666    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlfjm\"/\"kube-root-ca.crt\""
Sep 17 18:07:02.990408 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.990390    2521 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wlfjm\"/\"default-dockercfg-v2w9r\""
Sep 17 18:07:02.990506 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:02.990407    2521 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlfjm\"/\"openshift-service-ca.crt\""
Sep 17 18:07:03.003295 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.003270    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"]
Sep 17 18:07:03.098281 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.098250    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-podres\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.098430 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.098304    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-proc\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.098430 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.098337    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-lib-modules\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.098430 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.098364    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2q4s\" (UniqueName: \"kubernetes.io/projected/7b3183c0-1e62-4bcd-aca8-09028277dfe0-kube-api-access-d2q4s\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.098430 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.098392    2521 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-sys\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.199253 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.199220    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-podres\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.199376 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.199268    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-proc\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.199376 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.199287    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-lib-modules\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.199376 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.199307    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2q4s\" (UniqueName: \"kubernetes.io/projected/7b3183c0-1e62-4bcd-aca8-09028277dfe0-kube-api-access-d2q4s\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.199376 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.199333    2521 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-sys\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.199376 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.199369    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-proc\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.199592 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.199406    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-sys\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.199592 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.199408    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-podres\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.199592 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.199416    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b3183c0-1e62-4bcd-aca8-09028277dfe0-lib-modules\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.209246 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.209227    2521 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2q4s\" (UniqueName: \"kubernetes.io/projected/7b3183c0-1e62-4bcd-aca8-09028277dfe0-kube-api-access-d2q4s\") pod \"perf-node-gather-daemonset-wvtl8\" (UID: \"7b3183c0-1e62-4bcd-aca8-09028277dfe0\") " pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.296191 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.296120    2521 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:03.569067 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:03.568990    2521 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"]
Sep 17 18:07:03.572879 ip-10-0-135-179 kubenswrapper[2521]: W0917 18:07:03.572844    2521 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7b3183c0_1e62_4bcd_aca8_09028277dfe0.slice/crio-12c06ad976548ff311d470498502de6e5133925824abfafbf20e2a0c91ff8574 WatchSource:0}: Error finding container 12c06ad976548ff311d470498502de6e5133925824abfafbf20e2a0c91ff8574: Status 404 returned error can't find the container with id 12c06ad976548ff311d470498502de6e5133925824abfafbf20e2a0c91ff8574
Sep 17 18:07:04.408618 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:04.408576    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8" event={"ID":"7b3183c0-1e62-4bcd-aca8-09028277dfe0","Type":"ContainerStarted","Data":"1c4f390ff4407174966a75f6315b76da587311fbfbdeff3bb14e4826dfba819f"}
Sep 17 18:07:04.409019 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:04.408641    2521 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8" event={"ID":"7b3183c0-1e62-4bcd-aca8-09028277dfe0","Type":"ContainerStarted","Data":"12c06ad976548ff311d470498502de6e5133925824abfafbf20e2a0c91ff8574"}
Sep 17 18:07:04.409019 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:04.408682    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:04.439840 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:04.439793    2521 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8" podStartSLOduration=2.439778871 podStartE2EDuration="2.439778871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:04.438485089 +0000 UTC m=+1999.483490177" watchObservedRunningTime="2026-09-17 18:07:04.439778871 +0000 UTC m=+1999.484783959"
Sep 17 18:07:04.760543 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:04.760471    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6zpfd_058158ca-fba0-4467-9c85-4661b64588ee/dns/0.log"
Sep 17 18:07:04.799960 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:04.799935    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6zpfd_058158ca-fba0-4467-9c85-4661b64588ee/kube-rbac-proxy/0.log"
Sep 17 18:07:04.928161 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:04.928131    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-89wfr_3995ed8a-03ec-4c9a-b39f-014901ab9c47/dns-node-resolver/0.log"
Sep 17 18:07:05.571814 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:05.571768    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2rxr8_a18f3ab3-0d68-49b0-8189-1aa420dc674d/node-ca/0.log"
Sep 17 18:07:07.125565 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:07.125512    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9hzc9_daad7b8f-2beb-46aa-91d7-cafffb2e11e4/serve-healthcheck-canary/0.log"
Sep 17 18:07:08.012543 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:08.012510    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xf4jk_796e0187-1f6d-4306-bdd4-999bb67a1a21/kube-rbac-proxy/0.log"
Sep 17 18:07:08.047790 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:08.047768    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xf4jk_796e0187-1f6d-4306-bdd4-999bb67a1a21/exporter/0.log"
Sep 17 18:07:08.092354 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:08.092330    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xf4jk_796e0187-1f6d-4306-bdd4-999bb67a1a21/extractor/0.log"
Sep 17 18:07:10.420874 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:10.420846    2521 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wlfjm/perf-node-gather-daemonset-wvtl8"
Sep 17 18:07:10.521810 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:10.521772    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7f654675b-twgwm_15baa177-10a8-4f00-b3b9-dcef0efacbc9/manager/0.log"
Sep 17 18:07:10.580267 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:10.580238    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-vj2g2_7f25468d-fe42-4d6f-8c96-a29b3f0c1c5b/manager/0.log"
Sep 17 18:07:17.784840 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:17.784810    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9zzms_8c529cb3-09aa-42d0-ae31-9b76df0ab375/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:07:17.809371 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:17.809347    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9zzms_8c529cb3-09aa-42d0-ae31-9b76df0ab375/egress-router-binary-copy/0.log"
Sep 17 18:07:17.832120 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:17.832101    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9zzms_8c529cb3-09aa-42d0-ae31-9b76df0ab375/cni-plugins/0.log"
Sep 17 18:07:17.857062 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:17.857038    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9zzms_8c529cb3-09aa-42d0-ae31-9b76df0ab375/bond-cni-plugin/0.log"
Sep 17 18:07:17.883178 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:17.883158    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9zzms_8c529cb3-09aa-42d0-ae31-9b76df0ab375/routeoverride-cni/0.log"
Sep 17 18:07:17.907415 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:17.907389    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9zzms_8c529cb3-09aa-42d0-ae31-9b76df0ab375/whereabouts-cni-bincopy/0.log"
Sep 17 18:07:17.935997 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:17.935976    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9zzms_8c529cb3-09aa-42d0-ae31-9b76df0ab375/whereabouts-cni/0.log"
Sep 17 18:07:18.385181 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:18.385115    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tn7bn_6b7acfa2-1f97-45c6-934a-d9b9b6098c9d/kube-multus/0.log"
Sep 17 18:07:18.578188 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:18.578159    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n6dxl_e8594596-9c6f-4e0f-9892-21e06eba0cad/network-metrics-daemon/0.log"
Sep 17 18:07:18.620813 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:18.620788    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n6dxl_e8594596-9c6f-4e0f-9892-21e06eba0cad/kube-rbac-proxy/0.log"
Sep 17 18:07:19.428115 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:19.428080    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qk8j_c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6/ovn-controller/0.log"
Sep 17 18:07:19.450321 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:19.450296    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qk8j_c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6/ovn-acl-logging/0.log"
Sep 17 18:07:19.476759 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:19.476736    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qk8j_c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6/kube-rbac-proxy-node/0.log"
Sep 17 18:07:19.502760 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:19.502742    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qk8j_c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:07:19.525206 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:19.525185    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qk8j_c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6/northd/0.log"
Sep 17 18:07:19.548361 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:19.548344    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qk8j_c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6/nbdb/0.log"
Sep 17 18:07:19.571898 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:19.571876    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qk8j_c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6/sbdb/0.log"
Sep 17 18:07:19.612547 ip-10-0-135-179 kubenswrapper[2521]: I0917 18:07:19.612527    2521 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9qk8j_c4a91a86-eab6-46ef-b4ec-713c7bdb1bd6/ovnkube-controller/0.log"