Sep 09 19:55:27.008459 ip-10-0-143-121 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 19:55:27.421861 ip-10-0-143-121 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:27.421861 ip-10-0-143-121 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 19:55:27.421861 ip-10-0-143-121 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:27.421861 ip-10-0-143-121 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 19:55:27.421861 ip-10-0-143-121 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:27.423107 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.422879    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 19:55:27.427250 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427235    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:27.427250 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427251    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427255    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427258    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427261    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427264    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427267    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427272    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427276    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427279    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427281    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427284    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427288    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427290    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427293    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427296    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427299    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427302    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427305    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427307    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427310    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:27.427319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427313    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427315    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427318    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427321    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427324    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427327    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427330    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427333    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427335    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427338    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427341    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427343    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427346    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427349    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427351    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427354    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427357    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427359    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427362    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:27.427798 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427364    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427367    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427370    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427374    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427378    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427381    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427384    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427386    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427391    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427394    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427397    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427400    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427403    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427406    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427408    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427412    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427415    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427418    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427422    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:27.428279 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427426    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427429    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427432    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427434    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427437    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427439    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427442    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427444    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427447    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427450    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427452    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427454    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427457    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427460    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427464    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427466    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427469    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427472    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427474    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427477    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:27.428755 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427479    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427482    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427485    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427487    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427491    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427493    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427496    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427498    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427502    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427505    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427507    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427510    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427513    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427515    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427518    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427520    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427523    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427526    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427528    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.427531    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:27.429259 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429219    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429227    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429231    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429234    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429237    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429240    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429243    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429246    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429248    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429251    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429254    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429256    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429259    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429261    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429264    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429267    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429270    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429272    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429275    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429278    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:27.429762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429280    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429283    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429287    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429291    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429294    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429297    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429300    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429302    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429305    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429307    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429310    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429312    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429315    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429318    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429322    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429326    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429329    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429331    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429335    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:27.430289 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429337    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429340    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429343    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429345    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429348    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429351    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429353    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429356    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429358    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429361    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429364    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429367    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429369    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429372    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429374    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429377    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429380    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429382    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429385    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429388    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:27.430764 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429390    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429393    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429397    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429400    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429403    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429406    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429409    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429414    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429418    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429421    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429424    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429426    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429435    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429437    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429440    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429443    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429445    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429448    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429450    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:27.431302 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429453    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429455    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429458    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429461    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429463    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429466    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429468    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429471    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429473    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429476    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429478    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429481    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429483    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429485    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429488    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429490    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429493    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429496    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429498    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429501    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:27.431774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429504    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429898    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429902    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429905    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429907    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429911    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429913    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429916    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429919    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429921    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429924    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429927    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429929    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429932    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429934    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429937    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429939    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429942    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429945    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:27.432319 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429947    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429950    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429952    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429955    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429957    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429960    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429962    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429965    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429967    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429972    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429976    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429980    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429983    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429987    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429990    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429993    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429996    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.429998    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430001    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:27.432795 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430004    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430007    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430009    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430012    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430015    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430018    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430020    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430023    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430025    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430028    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430030    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430033    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430035    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430038    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430040    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430043    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430045    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430048    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430051    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:27.433277 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430054    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430058    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430061    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430066    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430069    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430071    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430074    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430077    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430080    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430083    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430085    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430088    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430091    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430093    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430096    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430099    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430101    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430103    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430106    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430109    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:27.433774 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430111    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430114    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430116    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430119    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430121    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430124    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430126    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430129    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430131    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430133    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430136    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430139    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430141    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430144    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430146    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430150    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430153    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430155    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430170    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430174    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:27.434330 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430176    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430179    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430182    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430252    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430284    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430291    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430296    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430300    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430304    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430308    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430312    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430315    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430318    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430321    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430325    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430328    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430331    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430334    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430337    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430339    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430342    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430346    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430349    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 19:55:27.434896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430352    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430355    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430359    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430362    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430366    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430369    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430372    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430375    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430378    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430381    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430384    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430387    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430391    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430394    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430397    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430400    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430403    2557 flags.go:64] FLAG: --enable-server="true"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430407    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430411    2557 flags.go:64] FLAG: --event-burst="100"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430414    2557 flags.go:64] FLAG: --event-qps="50"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430417    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430420    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430423    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430427    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 19:55:27.435702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430430    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430432    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430435    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430438    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430441    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430444    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430447    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430449    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430452    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430455    2557 flags.go:64] FLAG: --feature-gates=""
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430459    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430461    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430465    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430468    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430471    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430474    2557 flags.go:64] FLAG: --help="false"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430477    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430481    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430484    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430487    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430490    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430493    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430497    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 19:55:27.436384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430500    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430503    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430505    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430509    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430512    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430515    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430518    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430521    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430524    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430527    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430530    2557 flags.go:64] FLAG: --lock-file=""
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430533    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430535    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430538    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430544    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430546    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430549    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430552    2557 flags.go:64] FLAG: --logging-format="text"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430555    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430558    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430561    2557 flags.go:64] FLAG: --manifest-url=""
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430564    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430568    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430571    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430575    2557 flags.go:64] FLAG: --max-pods="110"
Sep 09 19:55:27.437015 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430578    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430581    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430584    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430587    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430590    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430593    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430596    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430603    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430606    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430609    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430612    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430615    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430620    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430623    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430626    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430629    2557 flags.go:64] FLAG: --port="10250"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430632    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430637    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-019db3a53daa8585b"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430640    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430643    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430645    2557 flags.go:64] FLAG: --register-node="true"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430648    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430652    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430654    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 09 19:55:27.437676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430657    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430660    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430663    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430666    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430669    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430672    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430675    2557 flags.go:64] FLAG: --runonce="false"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430678    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430681    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430684    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430686    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430689    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430692    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430695    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430698    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430700    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430704    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430706    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430709    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430713    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430716    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430719    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430724    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430727    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430729    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 19:55:27.438293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430734    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430737    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430739    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430742    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430745    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430748    2557 flags.go:64] FLAG: --v="2"
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430751    2557 flags.go:64] FLAG: --version="false"
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430755    2557 flags.go:64] FLAG: --vmodule=""
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430759    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.430762    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430876    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430880    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430884    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430887    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430890    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430893    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430895    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430898    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430901    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430903    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430906    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430910    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:27.438906 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430914    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430916    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430919    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430921    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430924    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430926    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430929    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430932    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430935    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430938    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430943    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430945    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430948    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430951    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430953    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430956    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430967    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430972    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430975    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:27.439530 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430977    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430980    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430983    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430986    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430989    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430992    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430995    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.430997    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431000    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431003    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431005    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431008    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431010    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431013    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431015    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431018    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431021    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431023    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431026    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431028    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:27.440006 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431032    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431034    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431037    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431041    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431044    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431048    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431051    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431054    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431057    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431059    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431062    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431065    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431068    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431070    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431073    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431076    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431079    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431082    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431084    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:27.440762 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431087    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431089    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431092    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431094    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431097    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431099    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431102    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431105    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431107    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431110    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431112    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431115    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431117    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431120    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431123    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431126    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431129    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431132    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431134    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431137    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:27.441410 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431139    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431142    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431144    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431147    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431150    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431152    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431155    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431171    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.431175    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.431861    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.439841    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.439860    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.439877    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.439883    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.439943    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:27.441896 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.439947    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.439950    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.439954    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.439958    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.439960    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441060    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441071    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441076    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441082    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441086    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441094    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441102    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441107    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441113    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441118    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441122    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441127    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441137    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441141    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:27.442303 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441212    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441234    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441239    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441243    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441247    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441251    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441254    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441258    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441276    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441301    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441306    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441316    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441456    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441461    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441465    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441468    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441471    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441475    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441479    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:27.442759 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441482    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441485    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441489    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441492    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441495    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441498    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441501    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441504    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441507    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441510    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441513    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441516    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441519    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441521    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441524    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441527    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441530    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441532    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441536    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441540    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:27.443349 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441542    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441545    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441547    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441550    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441553    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441556    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441558    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441561    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441566    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441569    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441573    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441575    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441578    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441582    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441585    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441587    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441590    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441592    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441595    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441599    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:27.443827 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441602    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441607    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441610    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441615    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441617    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441620    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441624    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441627    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441631    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441633    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441638    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441641    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441644    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441646    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441650    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441652    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441655    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441658    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441660    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:27.444335 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441663    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.441668    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441801    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441806    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441809    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441812    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441815    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441818    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441820    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441823    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441826    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441829    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441831    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441834    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441836    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:27.444786 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441839    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441841    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441844    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441846    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441849    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441851    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441854    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441856    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441859    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441862    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441865    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441867    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441870    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441873    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441875    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441878    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441880    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441883    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441886    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441889    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:27.445214 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441892    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441894    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441897    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441899    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441902    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441905    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441907    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441910    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441913    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441916    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441918    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441920    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441923    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441926    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441928    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441931    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441934    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441936    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441939    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:27.445706 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441943    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441947    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441950    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441955    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441958    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441961    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441963    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441966    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441968    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441971    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441973    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441976    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441978    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441981    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441983    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441986    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441989    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441991    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441993    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441996    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:27.446170 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.441999    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442002    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442004    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442006    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442009    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442011    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442014    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442016    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442019    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442021    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442025    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442029    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442032    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442035    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442038    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442040    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442046    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442049    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442052    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:27.446658 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442055    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442057    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442059    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442062    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442065    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442067    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442069    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:27.442073    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.442078    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.442222    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.444692    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.445713    2557 server.go:1019] "Starting client certificate rotation"
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.445813    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:27.447135 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.445841    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:27.448506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.448494    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:27.449627 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.449612    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:27.461868 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.461848    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 09 19:55:27.466066 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.466051    2557 log.go:25] "Validated CRI v1 image API"
Sep 09 19:55:27.466950 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.466936    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 19:55:27.470557 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.470539    2557 fs.go:135] Filesystem UUIDs: map[0a6d7750-db36-4ad9-b50d-417a5d5d51b7:/dev/nvme0n1p3 23603490-7efb-4478-a3d0-66ef440de528:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 09 19:55:27.470617 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.470556    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 19:55:27.477953 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.477577    2557 manager.go:216] Machine: {Timestamp:2026-09-09 19:55:27.476319641 +0000 UTC m=+0.372341462 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102713 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa6cc45611e7916e5163526db6d4c SystemUUID:ec2aa6cc-4561-1e79-16e5-163526db6d4c BootID:0fbf39ce-fd7f-4904-8a07-9fb43caa4d68 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:82:34:d2:84:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:82:34:d2:84:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:a9:81:12:ff:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 19:55:27.477953 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.477942    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 19:55:27.478113 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.478099    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 19:55:27.481721 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.481686    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 19:55:27.481858 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.481724    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 19:55:27.482601 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.482590    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 19:55:27.482643 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.482604    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 19:55:27.482643 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.482611    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:27.483550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.483539    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:27.484818 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.484801    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:27.485284 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.485272    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 19:55:27.490454 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.490442    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:27.490578 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.490569    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 19:55:27.490617 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.490598    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 19:55:27.490617 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.490609    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 19:55:27.490668 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.490630    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 19:55:27.490668 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.490647    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 09 19:55:27.490668 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.490657    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 19:55:27.491084 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.491073    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:27.491128 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.491090    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:27.493416 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.493398    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 19:55:27.496308 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.496290    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 19:55:27.496397 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.496329    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 19:55:27.496530 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.496517    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 19:55:27.496591 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.496535    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 19:55:27.496591 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.496544    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 19:55:27.496591 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.496553    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 19:55:27.496591 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.496564    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 19:55:27.496591 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.496573    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 19:55:27.497128 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.497118    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 19:55:27.497192 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.497130    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 19:55:27.497192 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.497141    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 19:55:27.497192 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.497149    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 19:55:27.497192 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.497179    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 19:55:27.497327 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.497201    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 19:55:27.498200 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.498190    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 19:55:27.498255 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.498203    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 19:55:27.500834 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.500805    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 19:55:27.501452 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.501433    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 19:55:27.501492 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.501434    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 19:55:27.502126 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.502115    2557 server.go:1268] "Started kubelet"
Sep 09 19:55:27.502263 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.502204    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 19:55:27.502316 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.502252    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 19:55:27.502316 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.502315    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 19:55:27.503199 ip-10-0-143-121 systemd[1]: Started Kubernetes Kubelet.
Sep 09 19:55:27.504228 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.504109    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 19:55:27.504347 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.504335    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 19:55:27.504857 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.504782    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnbjr"
Sep 09 19:55:27.508449 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.508421    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 19:55:27.508851 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.507727    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-121.ec2.internal.18d3bf38c2b75d05  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-121.ec2.internal,UID:ip-10-0-143-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-121.ec2.internal,},FirstTimestamp:2026-09-09 19:55:27.502089477 +0000 UTC m=+0.398111294,LastTimestamp:2026-09-09 19:55:27.502089477 +0000 UTC m=+0.398111294,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-121.ec2.internal,}"
Sep 09 19:55:27.509871 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.509857    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 19:55:27.509918 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.509869    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:27.510024 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.510002    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:27.510625 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.510609    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 19:55:27.510625 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.510627    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 19:55:27.511783 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.510996    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 19:55:27.511783 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.511019    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 19:55:27.511783 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.510806    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 19:55:27.512617 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.512595    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnbjr"
Sep 09 19:55:27.513195 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.513176    2557 factory.go:55] Registering systemd factory
Sep 09 19:55:27.513269 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.513220    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 09 19:55:27.513524 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.513506    2557 factory.go:153] Registering CRI-O factory
Sep 09 19:55:27.513595 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.513528    2557 factory.go:223] Registration of the crio container factory successfully
Sep 09 19:55:27.513595 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.513583    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 19:55:27.513680 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.513602    2557 factory.go:103] Registering Raw factory
Sep 09 19:55:27.513680 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.513619    2557 manager.go:1196] Started watching for new ooms in manager
Sep 09 19:55:27.514125 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.514110    2557 manager.go:349] Starting recovery of all containers
Sep 09 19:55:27.520021 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.520002    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:27.523088 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.523070    2557 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-121.ec2.internal\" not found" node="ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.525497 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.525480    2557 manager.go:354] Recovery completed
Sep 09 19:55:27.529605 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.529589    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:27.535457 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.535436    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:27.535542 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.535471    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:27.535542 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.535482    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:27.536091 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.536075    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 19:55:27.536091 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.536092    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 19:55:27.536219 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.536101    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:27.536219 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.536108    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:27.536913 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.536897    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:27.536974 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.536926    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:27.536974 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.536941    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:27.538257 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.538245    2557 policy_none.go:49] "None policy: Start"
Sep 09 19:55:27.538305 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.538261    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 19:55:27.538305 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.538273    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 19:55:27.540313 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.540299    2557 policy_none.go:47] "Start"
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.585786    2557 manager.go:337] "Starting Device Plugin manager"
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.585822    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.585834    2557 server.go:87] "Starting device plugin registration server"
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.586124    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.586138    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.586255    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.586329    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.586338    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.586878    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 19:55:27.589069 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.586931    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:27.631786 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.631743    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 19:55:27.632972 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.632950    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 19:55:27.633081 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.633007    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 19:55:27.633081 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.633031    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 19:55:27.633177 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.633079    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 19:55:27.635667 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.635649    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:27.686956 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.686845    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:27.687891 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.687872    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:27.687978 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.687904    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:27.687978 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.687918    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:27.687978 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.687943    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.696103 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.696082    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.696235 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.696112    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-121.ec2.internal\": node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:27.711508 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.711477    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:27.733332 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.733303    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal"]
Sep 09 19:55:27.733435 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.733372    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:27.734885 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.734866    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:27.734970 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.734896    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:27.734970 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.734907    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:27.737025 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.737013    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:27.737182 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.737152    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.737225 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.737197    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:27.737828 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.737812    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:27.737900 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.737838    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:27.737900 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.737853    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:27.737900 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.737816    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:27.738041 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.737925    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:27.738041 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.737943    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:27.740029 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.740014    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.740108 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.740042    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:27.741070 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.741055    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:27.741126 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.741080    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:27.741126 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.741092    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:27.768933 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.768911    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-121.ec2.internal\" not found" node="ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.773621 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.773599    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-121.ec2.internal\" not found" node="ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.812307 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.812280    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:27.812411 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.812350    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/efd24342e66e0564f736a8b8390c0964-config\") pod \"kube-apiserver-proxy-ip-10-0-143-121.ec2.internal\" (UID: \"efd24342e66e0564f736a8b8390c0964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.812411 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.812390    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dd7bc82853b4de2a4f6b51f0bebf3e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal\" (UID: \"8dd7bc82853b4de2a4f6b51f0bebf3e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.812476 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.812417    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dd7bc82853b4de2a4f6b51f0bebf3e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal\" (UID: \"8dd7bc82853b4de2a4f6b51f0bebf3e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.912458 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:27.912426    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:27.913573 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.913553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/efd24342e66e0564f736a8b8390c0964-config\") pod \"kube-apiserver-proxy-ip-10-0-143-121.ec2.internal\" (UID: \"efd24342e66e0564f736a8b8390c0964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.913614 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.913596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dd7bc82853b4de2a4f6b51f0bebf3e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal\" (UID: \"8dd7bc82853b4de2a4f6b51f0bebf3e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.913649 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.913611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dd7bc82853b4de2a4f6b51f0bebf3e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal\" (UID: \"8dd7bc82853b4de2a4f6b51f0bebf3e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.913649 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.913563    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/efd24342e66e0564f736a8b8390c0964-config\") pod \"kube-apiserver-proxy-ip-10-0-143-121.ec2.internal\" (UID: \"efd24342e66e0564f736a8b8390c0964\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.913714 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.913649    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dd7bc82853b4de2a4f6b51f0bebf3e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal\" (UID: \"8dd7bc82853b4de2a4f6b51f0bebf3e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:27.913714 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:27.913682    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dd7bc82853b4de2a4f6b51f0bebf3e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal\" (UID: \"8dd7bc82853b4de2a4f6b51f0bebf3e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:28.013179 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.013096    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.071405 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.071356    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:28.075990 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.075960    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:28.113836 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.113806    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.214228 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.214195    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.314317 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.314255    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.318484 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.318467    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:28.415048 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.415014    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.445227 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.445202    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 19:55:28.445604 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.445333    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:28.445604 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.445368    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:28.445604 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.445368    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:28.516200 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.516033    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.516200 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.516048    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 19:50:28 +0000 UTC" deadline="2028-03-10 10:33:25.614246919 +0000 UTC"
Sep 09 19:55:28.516389 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.516209    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13142h37m57.098043808s"
Sep 09 19:55:28.617217 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.617118    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.630525 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:28.630482    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd24342e66e0564f736a8b8390c0964.slice/crio-f9aa7601d0dd2e32164a1503bd91129f5a65396ad6d7fd71cb3f3dee55faff5f WatchSource:0}: Error finding container f9aa7601d0dd2e32164a1503bd91129f5a65396ad6d7fd71cb3f3dee55faff5f: Status 404 returned error can't find the container with id f9aa7601d0dd2e32164a1503bd91129f5a65396ad6d7fd71cb3f3dee55faff5f
Sep 09 19:55:28.634346 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.634327    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 19:55:28.636289 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.636249    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal" event={"ID":"efd24342e66e0564f736a8b8390c0964","Type":"ContainerStarted","Data":"f9aa7601d0dd2e32164a1503bd91129f5a65396ad6d7fd71cb3f3dee55faff5f"}
Sep 09 19:55:28.717316 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.717275    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.817752 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.817720    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.918235 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:28.918144    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-121.ec2.internal\" not found"
Sep 09 19:55:28.986853 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:28.986829    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:29.010432 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.010408    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:29.020563 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.020542    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:29.021331 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.021320    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal"
Sep 09 19:55:29.029249 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.029235    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:29.396670 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.396592    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:29.492263 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.492227    2557 apiserver.go:52] "Watching apiserver"
Sep 09 19:55:29.500223 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.499045    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 19:55:29.500223 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.499880    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-g9vl9","openshift-image-registry/node-ca-5d8wp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal","openshift-multus/multus-additional-cni-plugins-59wff","openshift-multus/network-metrics-daemon-2mfbb","kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rjfnj","openshift-multus/multus-f4pbr","openshift-network-diagnostics/network-check-target-dfq9t","openshift-network-operator/iptables-alerter-kn5c2","openshift-ovn-kubernetes/ovnkube-node-smhwd","kube-system/global-pull-secret-syncer-6jmkw","kube-system/konnectivity-agent-xzgsg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"]
Sep 09 19:55:29.502548 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.502525    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.504474 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.504454    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:29.504696 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.504673    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:29.504809 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.504518    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:29.504970 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.504952    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7hjhs\""
Sep 09 19:55:29.506304 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.506270    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 19:55:29.506580 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.506563    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n4hvk\""
Sep 09 19:55:29.506794 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.506769    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 19:55:29.507030 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.507007    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.508718 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.508699    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 19:55:29.508819 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.508728    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 19:55:29.508880 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.508868    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 19:55:29.508932 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.508876    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8v26\""
Sep 09 19:55:29.509109 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.509095    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.510406 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.510390    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:29.510895 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.510848    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 19:55:29.511134 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.511115    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:29.511248 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.511143    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-87b2c\""
Sep 09 19:55:29.511248 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.511182    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 19:55:29.511248 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.511204    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:29.511414 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.511268    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 19:55:29.511414 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.511373    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 19:55:29.511547 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.511439    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 19:55:29.513347 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.513329    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.515106 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.515087    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 19:55:29.515211 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.515124    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sfrps\""
Sep 09 19:55:29.515678 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.515656    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:29.515754 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.515731    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:29.517807 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.517789    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.519585 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.519565    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 19:55:29.519816 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.519796    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 19:55:29.520028 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520013    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b7ttl\""
Sep 09 19:55:29.520617 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-run-netns\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.520617 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520603    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-var-lib-cni-bin\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.520757 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520628    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-modprobe-d\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.520757 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520609    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.520757 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520651    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-sysconfig\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.520757 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520673    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-sys\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.520757 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520695    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshr8\" (UniqueName: \"kubernetes.io/projected/ca49d415-4424-4121-b375-b16f0d0abf34-kube-api-access-xshr8\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.520757 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520731    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-socket-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520759    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0d2e226-3707-4893-b20e-1686fddccf6b-system-cni-dir\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520783    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0d2e226-3707-4893-b20e-1686fddccf6b-cnibin\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520806    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-conf-dir\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520826    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-run\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520852    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca49d415-4424-4121-b375-b16f0d0abf34-etc-tuned\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpsr\" (UniqueName: \"kubernetes.io/projected/a0d2e226-3707-4893-b20e-1686fddccf6b-kube-api-access-ccpsr\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520898    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-cni-dir\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520938    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-cnibin\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520961    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-socket-dir-parent\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520986    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-run-k8s-cni-cncf-io\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521037    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-etc-kubernetes\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.521047 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.520852    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:29.521493 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-daemon-config\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.521493 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521137    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-var-lib-kubelet\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.521493 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-host\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.521632 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521212    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca49d415-4424-4121-b375-b16f0d0abf34-tmp\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.521687 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521650    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2-agent-certs\") pod \"konnectivity-agent-xzgsg\" (UID: \"d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2\") " pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:29.521687 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521676    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.521786 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521696    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-device-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.521786 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521718    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-kubernetes\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.521786 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521740    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-systemd\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.521786 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521780    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2-konnectivity-ca\") pod \"konnectivity-agent-xzgsg\" (UID: \"d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2\") " pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:29.521970 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0d2e226-3707-4893-b20e-1686fddccf6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.521970 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7x68\" (UniqueName: \"kubernetes.io/projected/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-kube-api-access-q7x68\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:29.521970 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521858    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-system-cni-dir\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.521970 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521885    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-var-lib-cni-multus\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.521970 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521912    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-sysctl-d\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.521970 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521943    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0d2e226-3707-4893-b20e-1686fddccf6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.521970    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-var-lib-kubelet\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522003    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-hostroot\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522024    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-sysctl-conf\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-lib-modules\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-etc-selinux\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-sys-fs\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522129    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xhq\" (UniqueName: \"kubernetes.io/projected/539f5db3-8f17-47ca-8ce1-afc3631a584e-kube-api-access-w2xhq\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0d2e226-3707-4893-b20e-1686fddccf6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522207    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522233    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-cni-binary-copy\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.522279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522256    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-run-multus-certs\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.522783 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522288    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrm2b\" (UniqueName: \"kubernetes.io/projected/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-kube-api-access-mrm2b\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.522783 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522325    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-registration-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.522783 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522348    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0d2e226-3707-4893-b20e-1686fddccf6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.522783 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522403    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mcv4q\""
Sep 09 19:55:29.522783 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522518    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 19:55:29.522783 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522608    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 19:55:29.523073 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522794    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 19:55:29.523073 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.522852    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.524609 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.524584    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:29.524698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.524609    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:29.524698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.524618    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-76dxj\""
Sep 09 19:55:29.524698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.524644    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 19:55:29.526596 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.526056    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.529007 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.528289    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 19:55:29.529007 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.528602    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:29.529007 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.528672    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:29.529007 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.528756    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bshc8\""
Sep 09 19:55:29.529007 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.528926    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 19:55:29.529469 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.529449    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 19:55:29.529550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.529519    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 19:55:29.529612 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.529554    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 19:55:29.529773 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.529748    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 19:55:29.540264 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.540244    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g49xc"
Sep 09 19:55:29.550217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.550196    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g49xc"
Sep 09 19:55:29.612258 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.612225    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 19:55:29.622784 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622755    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0d2e226-3707-4893-b20e-1686fddccf6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.622910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622790    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-var-lib-cni-bin\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.622910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622808    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshr8\" (UniqueName: \"kubernetes.io/projected/ca49d415-4424-4121-b375-b16f0d0abf34-kube-api-access-xshr8\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.622910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622834    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d393eee-6f0d-4ca1-bae0-2d669406772c-tmp-dir\") pod \"node-resolver-g9vl9\" (UID: \"7d393eee-6f0d-4ca1-bae0-2d669406772c\") " pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.622910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-etc-openvswitch\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.622910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622880    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-run-ovn\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.622910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622893    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-var-lib-cni-bin\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622924    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4176db56-adb0-4a49-bcd3-6720c27d4dbe-ovnkube-config\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622951    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-conf-dir\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.622981    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0bae31c-37ce-4f6c-8ece-a8f3be26cd45-iptables-alerter-script\") pod \"iptables-alerter-kn5c2\" (UID: \"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45\") " pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-slash\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623018    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-conf-dir\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623030    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4176db56-adb0-4a49-bcd3-6720c27d4dbe-ovn-node-metrics-cert\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpsr\" (UniqueName: \"kubernetes.io/projected/a0d2e226-3707-4893-b20e-1686fddccf6b-kube-api-access-ccpsr\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-socket-dir-parent\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623117    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvs8\" (UniqueName: \"kubernetes.io/projected/7d393eee-6f0d-4ca1-bae0-2d669406772c-kube-api-access-rdvs8\") pod \"node-resolver-g9vl9\" (UID: \"7d393eee-6f0d-4ca1-bae0-2d669406772c\") " pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623155    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-systemd-units\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.623217 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623195    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-daemon-config\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623270    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-socket-dir-parent\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jchvm\" (UniqueName: \"kubernetes.io/projected/e0bae31c-37ce-4f6c-8ece-a8f3be26cd45-kube-api-access-jchvm\") pod \"iptables-alerter-kn5c2\" (UID: \"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45\") " pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623365    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-device-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2-konnectivity-ca\") pod \"konnectivity-agent-xzgsg\" (UID: \"d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2\") " pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623442    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623446    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0d2e226-3707-4893-b20e-1686fddccf6b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-kubelet\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623493    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-device-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623519    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0d2e226-3707-4893-b20e-1686fddccf6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623547    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7x68\" (UniqueName: \"kubernetes.io/projected/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-kube-api-access-q7x68\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623570    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1fb14a2-9c6e-455e-a59a-bde2e283c026-kubelet-config\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623601    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-sysctl-conf\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623625    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-lib-modules\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623650    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec73c477-ff0a-4794-94d2-a905c2c7ba37-serviceca\") pod \"node-ca-5d8wp\" (UID: \"ec73c477-ff0a-4794-94d2-a905c2c7ba37\") " pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.623762 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623674    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-run-openvswitch\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623697    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-node-log\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623720    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-modprobe-d\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623739    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-daemon-config\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-sysconfig\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623741    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-sysctl-conf\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623806    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-sys\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623842    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-sys\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623840    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-etc-selinux\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-sys-fs\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623877    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-modprobe-d\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623894    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xhq\" (UniqueName: \"kubernetes.io/projected/539f5db3-8f17-47ca-8ce1-afc3631a584e-kube-api-access-w2xhq\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623813    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-sysconfig\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623910    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-lib-modules\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623929    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-etc-selinux\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623928    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0d2e226-3707-4893-b20e-1686fddccf6b-cni-binary-copy\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623963    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-sys-fs\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.624506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.623974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0d2e226-3707-4893-b20e-1686fddccf6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624006    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-cni-binary-copy\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-registration-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-run-netns\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624088    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1fb14a2-9c6e-455e-a59a-bde2e283c026-dbus\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624100    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-registration-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-run-multus-certs\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624150    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-run-netns\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624156    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0d2e226-3707-4893-b20e-1686fddccf6b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624149    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrm2b\" (UniqueName: \"kubernetes.io/projected/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-kube-api-access-mrm2b\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624215    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djllr\" (UniqueName: \"kubernetes.io/projected/4176db56-adb0-4a49-bcd3-6720c27d4dbe-kube-api-access-djllr\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624270    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-run-multus-certs\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-socket-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624365    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2-konnectivity-ca\") pod \"konnectivity-agent-xzgsg\" (UID: \"d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2\") " pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624364    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0d2e226-3707-4893-b20e-1686fddccf6b-system-cni-dir\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624407    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0d2e226-3707-4893-b20e-1686fddccf6b-cnibin\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624411    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0d2e226-3707-4893-b20e-1686fddccf6b-system-cni-dir\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.625097 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-cni-binary-copy\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624427    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-socket-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624449    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0d2e226-3707-4893-b20e-1686fddccf6b-cnibin\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624443    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-run\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca49d415-4424-4121-b375-b16f0d0abf34-etc-tuned\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624504    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clksw\" (UniqueName: \"kubernetes.io/projected/ec73c477-ff0a-4794-94d2-a905c2c7ba37-kube-api-access-clksw\") pod \"node-ca-5d8wp\" (UID: \"ec73c477-ff0a-4794-94d2-a905c2c7ba37\") " pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624528    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-run-ovn-kubernetes\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624484    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-run\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-cni-bin\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624574    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-cni-dir\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624623    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-cnibin\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624645    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-run-k8s-cni-cncf-io\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624668    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-etc-kubernetes\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624688    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-multus-cni-dir\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-run-netns\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-run-k8s-cni-cncf-io\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624738    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-etc-kubernetes\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.625955 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624777    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4176db56-adb0-4a49-bcd3-6720c27d4dbe-env-overrides\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624806    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4176db56-adb0-4a49-bcd3-6720c27d4dbe-ovnkube-script-lib\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624825    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-var-lib-kubelet\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624826    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-cnibin\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624847    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-host\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624831    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624886    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-var-lib-kubelet\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624884    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca49d415-4424-4121-b375-b16f0d0abf34-tmp\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624902    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-host\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2-agent-certs\") pod \"konnectivity-agent-xzgsg\" (UID: \"d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2\") " pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624932    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d393eee-6f0d-4ca1-bae0-2d669406772c-hosts-file\") pod \"node-resolver-g9vl9\" (UID: \"7d393eee-6f0d-4ca1-bae0-2d669406772c\") " pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624948    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-var-lib-openvswitch\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624964    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.624986    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-kubernetes\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625039    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/539f5db3-8f17-47ca-8ce1-afc3631a584e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-systemd\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625145    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-kubernetes\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.626698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625173    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0bae31c-37ce-4f6c-8ece-a8f3be26cd45-host-slash\") pod \"iptables-alerter-kn5c2\" (UID: \"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45\") " pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-log-socket\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625148    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-systemd\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-system-cni-dir\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625248    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-var-lib-cni-multus\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-sysctl-d\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625290    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-var-lib-cni-multus\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625297    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-cni-netd\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625292    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-system-cni-dir\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625351    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0d2e226-3707-4893-b20e-1686fddccf6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625382    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-var-lib-kubelet\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625386    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca49d415-4424-4121-b375-b16f0d0abf34-etc-sysctl-d\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-hostroot\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625431    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-host-var-lib-kubelet\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625450    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec73c477-ff0a-4794-94d2-a905c2c7ba37-host\") pod \"node-ca-5d8wp\" (UID: \"ec73c477-ff0a-4794-94d2-a905c2c7ba37\") " pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625465    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-hostroot\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625478    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625504    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-run-systemd\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.627565 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.625616    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:29.628461 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.625699    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs podName:31bdfd97-00d0-4a49-8b1f-8bd09ad7df87 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:30.125666713 +0000 UTC m=+3.021688539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs") pod "network-metrics-daemon-2mfbb" (UID: "31bdfd97-00d0-4a49-8b1f-8bd09ad7df87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:29.628461 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.625871    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0d2e226-3707-4893-b20e-1686fddccf6b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.628598 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.628576    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca49d415-4424-4121-b375-b16f0d0abf34-etc-tuned\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.628775 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.628760    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2-agent-certs\") pod \"konnectivity-agent-xzgsg\" (UID: \"d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2\") " pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:29.628940 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.628921    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca49d415-4424-4121-b375-b16f0d0abf34-tmp\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.638894 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.638469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshr8\" (UniqueName: \"kubernetes.io/projected/ca49d415-4424-4121-b375-b16f0d0abf34-kube-api-access-xshr8\") pod \"tuned-rjfnj\" (UID: \"ca49d415-4424-4121-b375-b16f0d0abf34\") " pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.638996 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.638905    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpsr\" (UniqueName: \"kubernetes.io/projected/a0d2e226-3707-4893-b20e-1686fddccf6b-kube-api-access-ccpsr\") pod \"multus-additional-cni-plugins-59wff\" (UID: \"a0d2e226-3707-4893-b20e-1686fddccf6b\") " pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.639075 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.639020    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7x68\" (UniqueName: \"kubernetes.io/projected/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-kube-api-access-q7x68\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:29.640735 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.640705    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xhq\" (UniqueName: \"kubernetes.io/projected/539f5db3-8f17-47ca-8ce1-afc3631a584e-kube-api-access-w2xhq\") pod \"aws-ebs-csi-driver-node-rv4qb\" (UID: \"539f5db3-8f17-47ca-8ce1-afc3631a584e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.640945 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.640925    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrm2b\" (UniqueName: \"kubernetes.io/projected/6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f-kube-api-access-mrm2b\") pod \"multus-f4pbr\" (UID: \"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f\") " pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.725936 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.725859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1fb14a2-9c6e-455e-a59a-bde2e283c026-dbus\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:29.725936 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.725895    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djllr\" (UniqueName: \"kubernetes.io/projected/4176db56-adb0-4a49-bcd3-6720c27d4dbe-kube-api-access-djllr\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.725936 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.725916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clksw\" (UniqueName: \"kubernetes.io/projected/ec73c477-ff0a-4794-94d2-a905c2c7ba37-kube-api-access-clksw\") pod \"node-ca-5d8wp\" (UID: \"ec73c477-ff0a-4794-94d2-a905c2c7ba37\") " pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.725938    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-run-ovn-kubernetes\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.725963    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-cni-bin\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.725994    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1fb14a2-9c6e-455e-a59a-bde2e283c026-dbus\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726008    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-cni-bin\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726021    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-run-ovn-kubernetes\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-run-netns\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726069    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-run-netns\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4176db56-adb0-4a49-bcd3-6720c27d4dbe-env-overrides\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726101    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4176db56-adb0-4a49-bcd3-6720c27d4dbe-ovnkube-script-lib\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d393eee-6f0d-4ca1-bae0-2d669406772c-hosts-file\") pod \"node-resolver-g9vl9\" (UID: \"7d393eee-6f0d-4ca1-bae0-2d669406772c\") " pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-var-lib-openvswitch\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726228    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726260    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0bae31c-37ce-4f6c-8ece-a8f3be26cd45-host-slash\") pod \"iptables-alerter-kn5c2\" (UID: \"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45\") " pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726269    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d393eee-6f0d-4ca1-bae0-2d669406772c-hosts-file\") pod \"node-resolver-g9vl9\" (UID: \"7d393eee-6f0d-4ca1-bae0-2d669406772c\") " pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-log-socket\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726324    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-var-lib-openvswitch\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726315    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-cni-netd\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726330    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726360    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec73c477-ff0a-4794-94d2-a905c2c7ba37-host\") pod \"node-ca-5d8wp\" (UID: \"ec73c477-ff0a-4794-94d2-a905c2c7ba37\") " pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726375    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0bae31c-37ce-4f6c-8ece-a8f3be26cd45-host-slash\") pod \"iptables-alerter-kn5c2\" (UID: \"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45\") " pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726387    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-log-socket\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726395    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-run-systemd\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726420    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-cni-netd\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d393eee-6f0d-4ca1-bae0-2d669406772c-tmp-dir\") pod \"node-resolver-g9vl9\" (UID: \"7d393eee-6f0d-4ca1-bae0-2d669406772c\") " pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726440    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-run-systemd\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-etc-openvswitch\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726486    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec73c477-ff0a-4794-94d2-a905c2c7ba37-host\") pod \"node-ca-5d8wp\" (UID: \"ec73c477-ff0a-4794-94d2-a905c2c7ba37\") " pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726488    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-run-ovn\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726512    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-run-ovn\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.726910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726521    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4176db56-adb0-4a49-bcd3-6720c27d4dbe-ovnkube-config\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726547    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0bae31c-37ce-4f6c-8ece-a8f3be26cd45-iptables-alerter-script\") pod \"iptables-alerter-kn5c2\" (UID: \"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45\") " pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726572    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-slash\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4176db56-adb0-4a49-bcd3-6720c27d4dbe-ovn-node-metrics-cert\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726548    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-etc-openvswitch\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726623    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4176db56-adb0-4a49-bcd3-6720c27d4dbe-env-overrides\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726648    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvs8\" (UniqueName: \"kubernetes.io/projected/7d393eee-6f0d-4ca1-bae0-2d669406772c-kube-api-access-rdvs8\") pod \"node-resolver-g9vl9\" (UID: \"7d393eee-6f0d-4ca1-bae0-2d669406772c\") " pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726691    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-systemd-units\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726681    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d393eee-6f0d-4ca1-bae0-2d669406772c-tmp-dir\") pod \"node-resolver-g9vl9\" (UID: \"7d393eee-6f0d-4ca1-bae0-2d669406772c\") " pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726717    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jchvm\" (UniqueName: \"kubernetes.io/projected/e0bae31c-37ce-4f6c-8ece-a8f3be26cd45-kube-api-access-jchvm\") pod \"iptables-alerter-kn5c2\" (UID: \"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45\") " pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726751    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-slash\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-systemd-units\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726796    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4176db56-adb0-4a49-bcd3-6720c27d4dbe-ovnkube-script-lib\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726865    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-kubelet\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726951    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1fb14a2-9c6e-455e-a59a-bde2e283c026-kubelet-config\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:29.727684 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.726989    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec73c477-ff0a-4794-94d2-a905c2c7ba37-serviceca\") pod \"node-ca-5d8wp\" (UID: \"ec73c477-ff0a-4794-94d2-a905c2c7ba37\") " pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.727013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-run-openvswitch\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.727036    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-node-log\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.727039    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4176db56-adb0-4a49-bcd3-6720c27d4dbe-ovnkube-config\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.727035    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.727085    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1fb14a2-9c6e-455e-a59a-bde2e283c026-kubelet-config\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.727110    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-node-log\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.727115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e0bae31c-37ce-4f6c-8ece-a8f3be26cd45-iptables-alerter-script\") pod \"iptables-alerter-kn5c2\" (UID: \"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45\") " pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.727121    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-run-openvswitch\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.727129    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret podName:b1fb14a2-9c6e-455e-a59a-bde2e283c026 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:30.227109884 +0000 UTC m=+3.123131706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret") pod "global-pull-secret-syncer-6jmkw" (UID: "b1fb14a2-9c6e-455e-a59a-bde2e283c026") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.727141    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4176db56-adb0-4a49-bcd3-6720c27d4dbe-host-kubelet\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.728324 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.727485    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ec73c477-ff0a-4794-94d2-a905c2c7ba37-serviceca\") pod \"node-ca-5d8wp\" (UID: \"ec73c477-ff0a-4794-94d2-a905c2c7ba37\") " pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.729369 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.729349    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4176db56-adb0-4a49-bcd3-6720c27d4dbe-ovn-node-metrics-cert\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.736291 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.736267    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djllr\" (UniqueName: \"kubernetes.io/projected/4176db56-adb0-4a49-bcd3-6720c27d4dbe-kube-api-access-djllr\") pod \"ovnkube-node-smhwd\" (UID: \"4176db56-adb0-4a49-bcd3-6720c27d4dbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.736387 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.736373    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clksw\" (UniqueName: \"kubernetes.io/projected/ec73c477-ff0a-4794-94d2-a905c2c7ba37-kube-api-access-clksw\") pod \"node-ca-5d8wp\" (UID: \"ec73c477-ff0a-4794-94d2-a905c2c7ba37\") " pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.742616 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.742594    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvs8\" (UniqueName: \"kubernetes.io/projected/7d393eee-6f0d-4ca1-bae0-2d669406772c-kube-api-access-rdvs8\") pod \"node-resolver-g9vl9\" (UID: \"7d393eee-6f0d-4ca1-bae0-2d669406772c\") " pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.743349 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.743327    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:29.743349 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.743354    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:29.743493 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.743367    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6m4pp for pod openshift-network-diagnostics/network-check-target-dfq9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:29.743493 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:29.743431    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp podName:1c4069a3-7ecc-4569-b976-3c59215327b2 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:30.243414074 +0000 UTC m=+3.139435883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6m4pp" (UniqueName: "kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp") pod "network-check-target-dfq9t" (UID: "1c4069a3-7ecc-4569-b976-3c59215327b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:29.744599 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.744578    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jchvm\" (UniqueName: \"kubernetes.io/projected/e0bae31c-37ce-4f6c-8ece-a8f3be26cd45-kube-api-access-jchvm\") pod \"iptables-alerter-kn5c2\" (UID: \"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45\") " pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.815222 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.815186    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjfnj"
Sep 09 19:55:29.822434 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.822409    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:29.833302 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.833281    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb"
Sep 09 19:55:29.838491 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.838470    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-59wff"
Sep 09 19:55:29.845149 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.845129    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f4pbr"
Sep 09 19:55:29.851705 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.851689    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9vl9"
Sep 09 19:55:29.858295 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.858279    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5d8wp"
Sep 09 19:55:29.865846 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.865829    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kn5c2"
Sep 09 19:55:29.870480 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.870462    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:29.969685 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:29.969659    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:30.129748 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.129650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:30.129909 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:30.129840    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:30.129953 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:30.129918    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs podName:31bdfd97-00d0-4a49-8b1f-8bd09ad7df87 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.129896391 +0000 UTC m=+4.025918212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs") pod "network-metrics-daemon-2mfbb" (UID: "31bdfd97-00d0-4a49-8b1f-8bd09ad7df87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:30.230364 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.230326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:30.230569 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:30.230493    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:30.230569 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:30.230567    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret podName:b1fb14a2-9c6e-455e-a59a-bde2e283c026 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.230550312 +0000 UTC m=+4.126572143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret") pod "global-pull-secret-syncer-6jmkw" (UID: "b1fb14a2-9c6e-455e-a59a-bde2e283c026") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:30.327639 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.327598    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d393eee_6f0d_4ca1_bae0_2d669406772c.slice/crio-809029800db9108f9f2f05669c2f5f081d7fcd0f3d4fdfb9db9e403383c60f6b WatchSource:0}: Error finding container 809029800db9108f9f2f05669c2f5f081d7fcd0f3d4fdfb9db9e403383c60f6b: Status 404 returned error can't find the container with id 809029800db9108f9f2f05669c2f5f081d7fcd0f3d4fdfb9db9e403383c60f6b
Sep 09 19:55:30.330769 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.330741    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:30.331000 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:30.330980    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:30.331094 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:30.331007    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:30.331094 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:30.331023    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6m4pp for pod openshift-network-diagnostics/network-check-target-dfq9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:30.331094 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:30.331089    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp podName:1c4069a3-7ecc-4569-b976-3c59215327b2 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.331068238 +0000 UTC m=+4.227090062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6m4pp" (UniqueName: "kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp") pod "network-check-target-dfq9t" (UID: "1c4069a3-7ecc-4569-b976-3c59215327b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:30.331559 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.331533    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea52b7e_e5d7_4053_b28d_7a9c6410dc1f.slice/crio-4cbb8caf144c36e2fd8646bd9b5b6589984245c05b59726c0dc7d3e60e9488d4 WatchSource:0}: Error finding container 4cbb8caf144c36e2fd8646bd9b5b6589984245c05b59726c0dc7d3e60e9488d4: Status 404 returned error can't find the container with id 4cbb8caf144c36e2fd8646bd9b5b6589984245c05b59726c0dc7d3e60e9488d4
Sep 09 19:55:30.332982 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.332959    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec73c477_ff0a_4794_94d2_a905c2c7ba37.slice/crio-f35037356c3e25a220385008e688ec179e5a58b10834642f68c861e794445dd0 WatchSource:0}: Error finding container f35037356c3e25a220385008e688ec179e5a58b10834642f68c861e794445dd0: Status 404 returned error can't find the container with id f35037356c3e25a220385008e688ec179e5a58b10834642f68c861e794445dd0
Sep 09 19:55:30.333887 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.333866    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bae31c_37ce_4f6c_8ece_a8f3be26cd45.slice/crio-dc89afed6d096fb66fed7f3eb439bdf0bf446a7daac32856fa2c1db4f037efa4 WatchSource:0}: Error finding container dc89afed6d096fb66fed7f3eb439bdf0bf446a7daac32856fa2c1db4f037efa4: Status 404 returned error can't find the container with id dc89afed6d096fb66fed7f3eb439bdf0bf446a7daac32856fa2c1db4f037efa4
Sep 09 19:55:30.334996 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.334920    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca49d415_4424_4121_b375_b16f0d0abf34.slice/crio-26f94d013e4ae665621d457e298ead1bced7d997d34513729f5f020cf690a618 WatchSource:0}: Error finding container 26f94d013e4ae665621d457e298ead1bced7d997d34513729f5f020cf690a618: Status 404 returned error can't find the container with id 26f94d013e4ae665621d457e298ead1bced7d997d34513729f5f020cf690a618
Sep 09 19:55:30.336516 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.336444    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539f5db3_8f17_47ca_8ce1_afc3631a584e.slice/crio-27072e609e239a1a8e238f4731b46d2a2798d3330264ff736dd597c9ddfbf796 WatchSource:0}: Error finding container 27072e609e239a1a8e238f4731b46d2a2798d3330264ff736dd597c9ddfbf796: Status 404 returned error can't find the container with id 27072e609e239a1a8e238f4731b46d2a2798d3330264ff736dd597c9ddfbf796
Sep 09 19:55:30.338453 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.338157    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29c4774_ef1c_46f7_b0c6_82ea8fdebaa2.slice/crio-a438d8e96225a2888fa76e7e9edc2d37e781f65abfcce3ff906ef7c83047fd06 WatchSource:0}: Error finding container a438d8e96225a2888fa76e7e9edc2d37e781f65abfcce3ff906ef7c83047fd06: Status 404 returned error can't find the container with id a438d8e96225a2888fa76e7e9edc2d37e781f65abfcce3ff906ef7c83047fd06
Sep 09 19:55:30.339623 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.338776    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d2e226_3707_4893_b20e_1686fddccf6b.slice/crio-86ea6ea618849a41aa3bc07d61111d688a84c6ad24363d859139011d631469b8 WatchSource:0}: Error finding container 86ea6ea618849a41aa3bc07d61111d688a84c6ad24363d859139011d631469b8: Status 404 returned error can't find the container with id 86ea6ea618849a41aa3bc07d61111d688a84c6ad24363d859139011d631469b8
Sep 09 19:55:30.339623 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.339576    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4176db56_adb0_4a49_bcd3_6720c27d4dbe.slice/crio-6def525b5b5ee1a77422437115b4e2393a7355dff9a1eddd9fdb468670924aa6 WatchSource:0}: Error finding container 6def525b5b5ee1a77422437115b4e2393a7355dff9a1eddd9fdb468670924aa6: Status 404 returned error can't find the container with id 6def525b5b5ee1a77422437115b4e2393a7355dff9a1eddd9fdb468670924aa6
Sep 09 19:55:30.507980 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:55:30.507955    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd7bc82853b4de2a4f6b51f0bebf3e3.slice/crio-57e15ed93d82fb3ddbd293d62f8ff9cfd52ee5b532a10b75ef7bf7c31145b0bf WatchSource:0}: Error finding container 57e15ed93d82fb3ddbd293d62f8ff9cfd52ee5b532a10b75ef7bf7c31145b0bf: Status 404 returned error can't find the container with id 57e15ed93d82fb3ddbd293d62f8ff9cfd52ee5b532a10b75ef7bf7c31145b0bf
Sep 09 19:55:30.554113 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.554081    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:30 +0000 UTC" deadline="2028-03-29 14:00:49.634513351 +0000 UTC"
Sep 09 19:55:30.554113 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.554110    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13602h5m19.080407131s"
Sep 09 19:55:30.644761 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.644688    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb" event={"ID":"539f5db3-8f17-47ca-8ce1-afc3631a584e","Type":"ContainerStarted","Data":"27072e609e239a1a8e238f4731b46d2a2798d3330264ff736dd597c9ddfbf796"}
Sep 09 19:55:30.646081 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.646046    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjfnj" event={"ID":"ca49d415-4424-4121-b375-b16f0d0abf34","Type":"ContainerStarted","Data":"26f94d013e4ae665621d457e298ead1bced7d997d34513729f5f020cf690a618"}
Sep 09 19:55:30.647239 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.647216    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kn5c2" event={"ID":"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45","Type":"ContainerStarted","Data":"dc89afed6d096fb66fed7f3eb439bdf0bf446a7daac32856fa2c1db4f037efa4"}
Sep 09 19:55:30.648362 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.648336    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5d8wp" event={"ID":"ec73c477-ff0a-4794-94d2-a905c2c7ba37","Type":"ContainerStarted","Data":"f35037356c3e25a220385008e688ec179e5a58b10834642f68c861e794445dd0"}
Sep 09 19:55:30.649484 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.649451    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9vl9" event={"ID":"7d393eee-6f0d-4ca1-bae0-2d669406772c","Type":"ContainerStarted","Data":"809029800db9108f9f2f05669c2f5f081d7fcd0f3d4fdfb9db9e403383c60f6b"}
Sep 09 19:55:30.651999 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.651973    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal" event={"ID":"efd24342e66e0564f736a8b8390c0964","Type":"ContainerStarted","Data":"d332dcc2d894d5f3fae65a229f8877f4297cc8f5bdfc8026cd52ad5262f03fed"}
Sep 09 19:55:30.653358 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.653338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal" event={"ID":"8dd7bc82853b4de2a4f6b51f0bebf3e3","Type":"ContainerStarted","Data":"57e15ed93d82fb3ddbd293d62f8ff9cfd52ee5b532a10b75ef7bf7c31145b0bf"}
Sep 09 19:55:30.654574 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.654554    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f4pbr" event={"ID":"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f","Type":"ContainerStarted","Data":"4cbb8caf144c36e2fd8646bd9b5b6589984245c05b59726c0dc7d3e60e9488d4"}
Sep 09 19:55:30.655822 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.655796    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" event={"ID":"4176db56-adb0-4a49-bcd3-6720c27d4dbe","Type":"ContainerStarted","Data":"6def525b5b5ee1a77422437115b4e2393a7355dff9a1eddd9fdb468670924aa6"}
Sep 09 19:55:30.657094 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.657073    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59wff" event={"ID":"a0d2e226-3707-4893-b20e-1686fddccf6b","Type":"ContainerStarted","Data":"86ea6ea618849a41aa3bc07d61111d688a84c6ad24363d859139011d631469b8"}
Sep 09 19:55:30.658321 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:30.658301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xzgsg" event={"ID":"d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2","Type":"ContainerStarted","Data":"a438d8e96225a2888fa76e7e9edc2d37e781f65abfcce3ff906ef7c83047fd06"}
Sep 09 19:55:31.065935 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:31.065766    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:31.138260 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:31.138207    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:31.138453 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.138420    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:31.138527 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.138488    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs podName:31bdfd97-00d0-4a49-8b1f-8bd09ad7df87 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.138469801 +0000 UTC m=+6.034491623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs") pod "network-metrics-daemon-2mfbb" (UID: "31bdfd97-00d0-4a49-8b1f-8bd09ad7df87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:31.239742 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:31.239700    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:31.239937 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.239921    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:31.240008 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.239995    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret podName:b1fb14a2-9c6e-455e-a59a-bde2e283c026 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.239971077 +0000 UTC m=+6.135992908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret") pod "global-pull-secret-syncer-6jmkw" (UID: "b1fb14a2-9c6e-455e-a59a-bde2e283c026") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:31.341066 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:31.340976    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:31.341274 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.341228    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:31.341274 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.341252    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:31.341274 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.341265    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6m4pp for pod openshift-network-diagnostics/network-check-target-dfq9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:31.341465 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.341340    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp podName:1c4069a3-7ecc-4569-b976-3c59215327b2 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.341319028 +0000 UTC m=+6.237340852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6m4pp" (UniqueName: "kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp") pod "network-check-target-dfq9t" (UID: "1c4069a3-7ecc-4569-b976-3c59215327b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:31.555257 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:31.555162    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:30 +0000 UTC" deadline="2028-05-22 21:19:14.304614004 +0000 UTC"
Sep 09 19:55:31.555257 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:31.555242    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14905h23m42.749376785s"
Sep 09 19:55:31.634041 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:31.633934    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:31.634200 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:31.633931    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:31.634200 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.634088    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:31.634327 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.634231    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:31.634327 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:31.634294    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:31.634451 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:31.634422    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:33.158886 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:33.158846    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:33.159543 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.159050    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:33.159543 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.159138    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs podName:31bdfd97-00d0-4a49-8b1f-8bd09ad7df87 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:37.159114782 +0000 UTC m=+10.055136606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs") pod "network-metrics-daemon-2mfbb" (UID: "31bdfd97-00d0-4a49-8b1f-8bd09ad7df87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:33.260033 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:33.259993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:33.260225 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.260187    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:33.260330 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.260277    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret podName:b1fb14a2-9c6e-455e-a59a-bde2e283c026 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:37.260255634 +0000 UTC m=+10.156277456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret") pod "global-pull-secret-syncer-6jmkw" (UID: "b1fb14a2-9c6e-455e-a59a-bde2e283c026") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:33.361586 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:33.360894    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:33.361586 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.361104    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:33.361586 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.361128    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:33.361586 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.361143    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6m4pp for pod openshift-network-diagnostics/network-check-target-dfq9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:33.361586 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.361226    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp podName:1c4069a3-7ecc-4569-b976-3c59215327b2 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:37.36119299 +0000 UTC m=+10.257214815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6m4pp" (UniqueName: "kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp") pod "network-check-target-dfq9t" (UID: "1c4069a3-7ecc-4569-b976-3c59215327b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:33.635543 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:33.635377    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:33.635543 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.635520    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:33.636437 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:33.635947    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:33.636437 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.636067    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:33.636437 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:33.636290    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:33.636437 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:33.636377    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:35.634806 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:35.634349    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:35.634806 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:35.634508    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:35.634806 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:35.634557    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:35.634806 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:35.634609    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:35.634806 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:35.634691    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:35.634806 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:35.634763    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:37.195750 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:37.195059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:37.195750 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.195273    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:37.195750 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.195346    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs podName:31bdfd97-00d0-4a49-8b1f-8bd09ad7df87 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:45.195325188 +0000 UTC m=+18.091347021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs") pod "network-metrics-daemon-2mfbb" (UID: "31bdfd97-00d0-4a49-8b1f-8bd09ad7df87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:37.296505 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:37.296459    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:37.296697 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.296615    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:37.296795 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.296699    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret podName:b1fb14a2-9c6e-455e-a59a-bde2e283c026 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:45.296682242 +0000 UTC m=+18.192704053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret") pod "global-pull-secret-syncer-6jmkw" (UID: "b1fb14a2-9c6e-455e-a59a-bde2e283c026") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:37.397499 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:37.397452    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:37.397693 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.397663    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:37.397767 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.397695    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:37.397767 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.397710    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6m4pp for pod openshift-network-diagnostics/network-check-target-dfq9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:37.397871 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.397781    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp podName:1c4069a3-7ecc-4569-b976-3c59215327b2 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:45.397758839 +0000 UTC m=+18.293780671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6m4pp" (UniqueName: "kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp") pod "network-check-target-dfq9t" (UID: "1c4069a3-7ecc-4569-b976-3c59215327b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:37.634131 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:37.633998    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:37.634131 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:37.634010    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:37.635196 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.635158    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:37.635345 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:37.635296    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:37.635416 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.635349    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:37.635416 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:37.635394    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:39.634418 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:39.634149    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:39.635102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:39.634157    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:39.635102 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:39.634535    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:39.635102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:39.634227    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:39.635102 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:39.634628    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:39.635102 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:39.634684    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:41.636871 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:41.636841    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:41.637273 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:41.636841    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:41.637273 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:41.636962    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:41.637273 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:41.636841    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:41.637273 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:41.637037    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:41.637273 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:41.637153    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:43.633767 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:43.633726    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:43.634204 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:43.633733    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:43.634204 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:43.633845    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:43.634204 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:43.633861    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:43.634204 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:43.633963    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:43.634204 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:43.634056    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:45.258916 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:45.258880    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:45.259426 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.259012    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:45.259426 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.259068    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs podName:31bdfd97-00d0-4a49-8b1f-8bd09ad7df87 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.259054285 +0000 UTC m=+34.155076093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs") pod "network-metrics-daemon-2mfbb" (UID: "31bdfd97-00d0-4a49-8b1f-8bd09ad7df87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:45.359648 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:45.359623    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:45.359781 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.359755    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:45.359833 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.359814    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret podName:b1fb14a2-9c6e-455e-a59a-bde2e283c026 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.35979974 +0000 UTC m=+34.255821545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret") pod "global-pull-secret-syncer-6jmkw" (UID: "b1fb14a2-9c6e-455e-a59a-bde2e283c026") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:45.459965 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:45.459930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:45.460137 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.460101    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:45.460137 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.460123    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:45.460137 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.460135    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6m4pp for pod openshift-network-diagnostics/network-check-target-dfq9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:45.460324 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.460215    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp podName:1c4069a3-7ecc-4569-b976-3c59215327b2 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.460195376 +0000 UTC m=+34.356217201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6m4pp" (UniqueName: "kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp") pod "network-check-target-dfq9t" (UID: "1c4069a3-7ecc-4569-b976-3c59215327b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:45.637154 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:45.637035    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:45.637328 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:45.637148    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:45.637328 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:45.637189    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:45.637328 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.637172    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:45.637328 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.637276    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:45.637531 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:45.637376    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:47.634023 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:47.633988    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:47.634454 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:47.634077    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:47.634454 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:47.634185    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:47.634454 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:47.634357    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:47.634454 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:47.634400    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:47.634625 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:47.634474    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:49.633542 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.633510    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:49.633542 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.633540    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:49.634127 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.633540    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:49.634127 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:49.633655    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:49.634127 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:49.633713    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:49.634127 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:49.633803    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:49.717811 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.717414    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" event={"ID":"4176db56-adb0-4a49-bcd3-6720c27d4dbe","Type":"ContainerStarted","Data":"6bb607fe3b047bb70fe28e4d5800d7e4d9b1629472e8c173b78471ac65f188a2"}
Sep 09 19:55:49.717811 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.717715    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" event={"ID":"4176db56-adb0-4a49-bcd3-6720c27d4dbe","Type":"ContainerStarted","Data":"ba18c9a7a4fa7007f27566ab0a78cf593e67db9bf58c7a06f0279809a291458e"}
Sep 09 19:55:49.720137 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.720108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjfnj" event={"ID":"ca49d415-4424-4121-b375-b16f0d0abf34","Type":"ContainerStarted","Data":"cf32616f6389be62c9981de6135ac3269561158aae646a53ab314c179cbaba1d"}
Sep 09 19:55:49.722518 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.722485    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f4pbr" event={"ID":"6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f","Type":"ContainerStarted","Data":"dd0c0f041600dcf78254a97f823b12e760dcd0a3cf866c71711266e82c49933e"}
Sep 09 19:55:49.738655 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.738269    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-121.ec2.internal" podStartSLOduration=20.738248598 podStartE2EDuration="20.738248598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:30.666798873 +0000 UTC m=+3.562820703" watchObservedRunningTime="2026-09-09 19:55:49.738248598 +0000 UTC m=+22.634270429"
Sep 09 19:55:49.738655 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.738356    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rjfnj" podStartSLOduration=2.630194202 podStartE2EDuration="21.738347185s" totalImagesPullingTime="19.108152983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:49.737717739 +0000 UTC m=+22.633739567" watchObservedRunningTime="2026-09-09 19:55:49.738347185 +0000 UTC m=+22.634369013"
Sep 09 19:55:49.754780 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:49.754729    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f4pbr" podStartSLOduration=2.590237283 podStartE2EDuration="21.754709841s" totalImagesPullingTime="19.164472558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:49.754022183 +0000 UTC m=+22.650044010" watchObservedRunningTime="2026-09-09 19:55:49.754709841 +0000 UTC m=+22.650731669"
Sep 09 19:55:50.726924 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.726829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5d8wp" event={"ID":"ec73c477-ff0a-4794-94d2-a905c2c7ba37","Type":"ContainerStarted","Data":"b77ea53b7ff19180289bc2aa6bcbf4ddf69d93aae4958ef33754b0dd0f0796a8"}
Sep 09 19:55:50.728656 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.728616    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9vl9" event={"ID":"7d393eee-6f0d-4ca1-bae0-2d669406772c","Type":"ContainerStarted","Data":"26e85c32ad9a64f3948828232613cbdf64d06b96fe9a89e8b95ed70c0ae8bb65"}
Sep 09 19:55:50.730387 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.730365    2557 generic.go:356] "Generic (PLEG): container finished" podID="8dd7bc82853b4de2a4f6b51f0bebf3e3" containerID="74156661a1bc482fb1e72803b92bea8564e30f21355d2e45e2291a49ae06a5e5" exitCode=0
Sep 09 19:55:50.730494 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.730434    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal" event={"ID":"8dd7bc82853b4de2a4f6b51f0bebf3e3","Type":"ContainerDied","Data":"74156661a1bc482fb1e72803b92bea8564e30f21355d2e45e2291a49ae06a5e5"}
Sep 09 19:55:50.733951 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.733928    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" event={"ID":"4176db56-adb0-4a49-bcd3-6720c27d4dbe","Type":"ContainerStarted","Data":"60e876c7a6a76cb468bc2b3f405a331a383432ea99ad39de15b0b2009315ce52"}
Sep 09 19:55:50.734052 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.733998    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" event={"ID":"4176db56-adb0-4a49-bcd3-6720c27d4dbe","Type":"ContainerStarted","Data":"30ed6bce3ad5e09cb34c4ef895fa69ccc098a2a3dba6184d44d99c26551598c0"}
Sep 09 19:55:50.734052 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.734044    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" event={"ID":"4176db56-adb0-4a49-bcd3-6720c27d4dbe","Type":"ContainerStarted","Data":"565a412e5f727460dd1cda425da81f34537cd439dd35906014f6ffaa60abbec9"}
Sep 09 19:55:50.734136 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.734075    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" event={"ID":"4176db56-adb0-4a49-bcd3-6720c27d4dbe","Type":"ContainerStarted","Data":"083e77ae1693aa20a82fa0ecece7fe8c16fa7b98c650c4299f0d2f222fb2295e"}
Sep 09 19:55:50.735529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.735508    2557 generic.go:356] "Generic (PLEG): container finished" podID="a0d2e226-3707-4893-b20e-1686fddccf6b" containerID="9a06917dd7ddfc99de4d5d3211f768ec41004858883fc2ea662d4f92be9b2d48" exitCode=0
Sep 09 19:55:50.735626 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.735597    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59wff" event={"ID":"a0d2e226-3707-4893-b20e-1686fddccf6b","Type":"ContainerDied","Data":"9a06917dd7ddfc99de4d5d3211f768ec41004858883fc2ea662d4f92be9b2d48"}
Sep 09 19:55:50.737470 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.737435    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xzgsg" event={"ID":"d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2","Type":"ContainerStarted","Data":"d6992d8697d64edba259ff80e360d5d63f267ae66175731875f2d8df328732bc"}
Sep 09 19:55:50.739010 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.738989    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb" event={"ID":"539f5db3-8f17-47ca-8ce1-afc3631a584e","Type":"ContainerStarted","Data":"ab77cc4724868a37d4d514676e1a5ed3048303e212cdbec0fc39cc19d14ac345"}
Sep 09 19:55:50.741876 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.741840    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5d8wp" podStartSLOduration=3.631164982 podStartE2EDuration="22.741828553s" totalImagesPullingTime="19.110663571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:50.741435438 +0000 UTC m=+23.637457267" watchObservedRunningTime="2026-09-09 19:55:50.741828553 +0000 UTC m=+23.637850413"
Sep 09 19:55:50.796446 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.796402    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xzgsg" podStartSLOduration=3.71487353 podStartE2EDuration="22.79638949s" totalImagesPullingTime="19.08151596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:50.79629162 +0000 UTC m=+23.692313448" watchObservedRunningTime="2026-09-09 19:55:50.79638949 +0000 UTC m=+23.692411322"
Sep 09 19:55:50.811719 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:50.811686    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g9vl9" podStartSLOduration=3.6828474829999998 podStartE2EDuration="22.81167459s" totalImagesPullingTime="19.128827107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:50.811649602 +0000 UTC m=+23.707671420" watchObservedRunningTime="2026-09-09 19:55:50.81167459 +0000 UTC m=+23.707696416"
Sep 09 19:55:51.258448 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.258424    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 19:55:51.598351 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.598039    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T19:55:51.258444134Z","UUID":"2defbcc6-fb1c-4e68-86d1-d0e00a24381e","Handler":null,"Name":"","Endpoint":""}
Sep 09 19:55:51.599926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.599900    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 19:55:51.599926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.599934    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 19:55:51.634350 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.634322    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:51.634506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.634323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:51.634506 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:51.634434    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:51.634506 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.634443    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:51.634634 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:51.634537    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:51.634681 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:51.634651    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:51.743615 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.743571    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal" event={"ID":"8dd7bc82853b4de2a4f6b51f0bebf3e3","Type":"ContainerStarted","Data":"4ee16e5b48b9502316cdfba3509afb9073f7dab79ffe96841f322ba7923cc526"}
Sep 09 19:55:51.745796 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.745766    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb" event={"ID":"539f5db3-8f17-47ca-8ce1-afc3631a584e","Type":"ContainerStarted","Data":"005a118a938b9ac8189fa037846c474d77fba1df4a71fab20187108ff6942b3c"}
Sep 09 19:55:51.747595 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.747569    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kn5c2" event={"ID":"e0bae31c-37ce-4f6c-8ece-a8f3be26cd45","Type":"ContainerStarted","Data":"ff04fd3f4e029cb5c4d800dbb0b8d1c9dce4d91e6a8162de938e52f32d6389df"}
Sep 09 19:55:51.757954 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.757904    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-121.ec2.internal" podStartSLOduration=22.757887464 podStartE2EDuration="22.757887464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:51.757724316 +0000 UTC m=+24.653746145" watchObservedRunningTime="2026-09-09 19:55:51.757887464 +0000 UTC m=+24.653909294"
Sep 09 19:55:51.770947 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:51.770912    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kn5c2" podStartSLOduration=4.655869394 podStartE2EDuration="23.770899803s" totalImagesPullingTime="19.115030409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:51.770758542 +0000 UTC m=+24.666780364" watchObservedRunningTime="2026-09-09 19:55:51.770899803 +0000 UTC m=+24.666921630"
Sep 09 19:55:52.754420 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:52.754337    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" event={"ID":"4176db56-adb0-4a49-bcd3-6720c27d4dbe","Type":"ContainerStarted","Data":"a131f5f3c85c4df2202b6e8501aced1c24311158e6d413f3f38c5e1713d9eb7c"}
Sep 09 19:55:53.633605 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:53.633575    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:53.633605 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:53.633608    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:53.633864 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:53.633579    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:53.633864 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:53.633685    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:53.633864 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:53.633803    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:53.634005 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:53.633894    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:53.802966 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:53.802933    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:53.804150 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:53.804018    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:55.634242 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.634042    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:55.634862 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.634042    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:55.634862 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:55.634317    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:55.634862 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.634093    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:55.634862 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:55.634383    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:55.634862 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:55.634472    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:55.764254 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.764152    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" event={"ID":"4176db56-adb0-4a49-bcd3-6720c27d4dbe","Type":"ContainerStarted","Data":"26701312f3cfca1bb0690f98cd20459d18af67f795edc135e1c1a9f570d9650c"}
Sep 09 19:55:55.764419 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.764401    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:55.766146 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.766120    2557 generic.go:356] "Generic (PLEG): container finished" podID="a0d2e226-3707-4893-b20e-1686fddccf6b" containerID="ca8e3d7c8dac58daeeb7cb80b1ca12d74d5b004b20ad322bbfd67a0ccff6c8d5" exitCode=0
Sep 09 19:55:55.766269 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.766217    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59wff" event={"ID":"a0d2e226-3707-4893-b20e-1686fddccf6b","Type":"ContainerDied","Data":"ca8e3d7c8dac58daeeb7cb80b1ca12d74d5b004b20ad322bbfd67a0ccff6c8d5"}
Sep 09 19:55:55.768534 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.768512    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb" event={"ID":"539f5db3-8f17-47ca-8ce1-afc3631a584e","Type":"ContainerStarted","Data":"aef9cec43039221773fd83e83276fe2f9f147bd799d57062028f816b1f76137c"}
Sep 09 19:55:55.779830 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.779812    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:55.811887 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.811836    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd" podStartSLOduration=8.674888814 podStartE2EDuration="27.81182321s" totalImagesPullingTime="19.136934396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:55.811769471 +0000 UTC m=+28.707791292" watchObservedRunningTime="2026-09-09 19:55:55.81182321 +0000 UTC m=+28.707845037"
Sep 09 19:55:55.846526 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:55.846487    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rv4qb" podStartSLOduration=3.538232526 podStartE2EDuration="27.846474546s" totalImagesPullingTime="24.30824202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:55.846362568 +0000 UTC m=+28.742384396" watchObservedRunningTime="2026-09-09 19:55:55.846474546 +0000 UTC m=+28.742496373"
Sep 09 19:55:56.772564 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:56.772484    2557 generic.go:356] "Generic (PLEG): container finished" podID="a0d2e226-3707-4893-b20e-1686fddccf6b" containerID="661780aa6ab1275f8a7123de0d5248223c81fb406502aebd00d109f089c28efe" exitCode=0
Sep 09 19:55:56.772564 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:56.772553    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59wff" event={"ID":"a0d2e226-3707-4893-b20e-1686fddccf6b","Type":"ContainerDied","Data":"661780aa6ab1275f8a7123de0d5248223c81fb406502aebd00d109f089c28efe"}
Sep 09 19:55:56.773520 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:56.773309    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:56.773520 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:56.773381    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:56.788494 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:56.788474    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:55:57.212594 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:57.212272    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6jmkw"]
Sep 09 19:55:57.212738 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:57.212664    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:57.212807 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:57.212784    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:57.217772 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:57.217730    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2mfbb"]
Sep 09 19:55:57.217944 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:57.217924    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:57.218075 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:57.218048    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:57.218495 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:57.218468    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dfq9t"]
Sep 09 19:55:57.218614 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:57.218563    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:57.218676 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:57.218636    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:57.776982 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:57.776903    2557 generic.go:356] "Generic (PLEG): container finished" podID="a0d2e226-3707-4893-b20e-1686fddccf6b" containerID="bb709dfc2947bbd9b6e2c8199fc6805713b4007ecfefc03385d1609a863cee94" exitCode=0
Sep 09 19:55:57.777325 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:57.776990    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59wff" event={"ID":"a0d2e226-3707-4893-b20e-1686fddccf6b","Type":"ContainerDied","Data":"bb709dfc2947bbd9b6e2c8199fc6805713b4007ecfefc03385d1609a863cee94"}
Sep 09 19:55:58.634004 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:58.633963    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:55:58.634187 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:58.634017    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:55:58.634187 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:58.634077    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:55:58.634325 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:58.634250    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:55:59.634407 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:59.634328    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:55:59.634848 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:55:59.634446    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:55:59.693120 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:59.693091    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:55:59.693304 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:59.693244    2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 19:55:59.693781 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:55:59.693757    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xzgsg"
Sep 09 19:56:00.633659 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:00.633629    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:56:00.633825 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:00.633629    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:56:00.633825 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:00.633735    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:56:00.633956 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:00.633830    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:56:01.272949 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:01.272920    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:56:01.273407 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:01.273064    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:56:01.273407 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:01.273142    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs podName:31bdfd97-00d0-4a49-8b1f-8bd09ad7df87 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:33.273122585 +0000 UTC m=+66.169144389 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs") pod "network-metrics-daemon-2mfbb" (UID: "31bdfd97-00d0-4a49-8b1f-8bd09ad7df87") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:56:01.374298 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:01.374259    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:56:01.374475 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:01.374442    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:56:01.374539 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:01.374513    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret podName:b1fb14a2-9c6e-455e-a59a-bde2e283c026 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:33.374495207 +0000 UTC m=+66.270517017 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret") pod "global-pull-secret-syncer-6jmkw" (UID: "b1fb14a2-9c6e-455e-a59a-bde2e283c026") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:56:01.474839 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:01.474790    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:56:01.475013 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:01.474915    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:56:01.475013 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:01.474936    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:56:01.475013 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:01.474947    2557 projected.go:196] Error preparing data for projected volume kube-api-access-6m4pp for pod openshift-network-diagnostics/network-check-target-dfq9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:56:01.475206 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:01.475011    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp podName:1c4069a3-7ecc-4569-b976-3c59215327b2 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:33.47499283 +0000 UTC m=+66.371014638 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6m4pp" (UniqueName: "kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp") pod "network-check-target-dfq9t" (UID: "1c4069a3-7ecc-4569-b976-3c59215327b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:56:01.633497 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:01.633403    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:56:01.633655 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:01.633545    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6jmkw" podUID="b1fb14a2-9c6e-455e-a59a-bde2e283c026"
Sep 09 19:56:02.634120 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:02.634089    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:56:02.634602 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:02.634525    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfq9t" podUID="1c4069a3-7ecc-4569-b976-3c59215327b2"
Sep 09 19:56:02.636538 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:02.634997    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:56:02.636538 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:02.635186    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2mfbb" podUID="31bdfd97-00d0-4a49-8b1f-8bd09ad7df87"
Sep 09 19:56:02.791814 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:02.791783    2557 generic.go:356] "Generic (PLEG): container finished" podID="a0d2e226-3707-4893-b20e-1686fddccf6b" containerID="6753dcb60a4d15a5ae5909837f440ff10ed08a9c40483e27e5a7dd638b200f5c" exitCode=0
Sep 09 19:56:02.791975 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:02.791841    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59wff" event={"ID":"a0d2e226-3707-4893-b20e-1686fddccf6b","Type":"ContainerDied","Data":"6753dcb60a4d15a5ae5909837f440ff10ed08a9c40483e27e5a7dd638b200f5c"}
Sep 09 19:56:03.340437 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.340216    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-121.ec2.internal" event="NodeReady"
Sep 09 19:56:03.340599 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.340468    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 19:56:03.385150 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.385120    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sm8l8"]
Sep 09 19:56:03.401006 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.400980    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-425sq"]
Sep 09 19:56:03.401156 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.401113    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.403056 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.403038    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-28vq4\""
Sep 09 19:56:03.403181 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.403079    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 19:56:03.403181 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.403091    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 19:56:03.414990 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.414968    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sm8l8"]
Sep 09 19:56:03.415084 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.415002    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-425sq"]
Sep 09 19:56:03.415122 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.415081    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-425sq"
Sep 09 19:56:03.417056 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.417034    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vnftv\""
Sep 09 19:56:03.417218 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.417200    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 19:56:03.417358 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.417220    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 19:56:03.417432 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.417420    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 19:56:03.489844 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.489816    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c0e6cc9-602e-4752-92b2-e32a62763c2a-tmp-dir\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.489844 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.489849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwlr\" (UniqueName: \"kubernetes.io/projected/2c0e6cc9-602e-4752-92b2-e32a62763c2a-kube-api-access-dtwlr\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.490053 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.489867    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab31a95f-faaf-45ea-9290-95198e492b95-cert\") pod \"ingress-canary-425sq\" (UID: \"ab31a95f-faaf-45ea-9290-95198e492b95\") " pod="openshift-ingress-canary/ingress-canary-425sq"
Sep 09 19:56:03.490053 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.489884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxxz\" (UniqueName: \"kubernetes.io/projected/ab31a95f-faaf-45ea-9290-95198e492b95-kube-api-access-xcxxz\") pod \"ingress-canary-425sq\" (UID: \"ab31a95f-faaf-45ea-9290-95198e492b95\") " pod="openshift-ingress-canary/ingress-canary-425sq"
Sep 09 19:56:03.490053 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.489930    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.490053 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.489994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0e6cc9-602e-4752-92b2-e32a62763c2a-config-volume\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.591116 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.591051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0e6cc9-602e-4752-92b2-e32a62763c2a-config-volume\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.591116 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.591099    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c0e6cc9-602e-4752-92b2-e32a62763c2a-tmp-dir\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.591328 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.591123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwlr\" (UniqueName: \"kubernetes.io/projected/2c0e6cc9-602e-4752-92b2-e32a62763c2a-kube-api-access-dtwlr\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.591328 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.591146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab31a95f-faaf-45ea-9290-95198e492b95-cert\") pod \"ingress-canary-425sq\" (UID: \"ab31a95f-faaf-45ea-9290-95198e492b95\") " pod="openshift-ingress-canary/ingress-canary-425sq"
Sep 09 19:56:03.591328 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.591187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxxz\" (UniqueName: \"kubernetes.io/projected/ab31a95f-faaf-45ea-9290-95198e492b95-kube-api-access-xcxxz\") pod \"ingress-canary-425sq\" (UID: \"ab31a95f-faaf-45ea-9290-95198e492b95\") " pod="openshift-ingress-canary/ingress-canary-425sq"
Sep 09 19:56:03.591328 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.591210    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.591328 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:03.591305    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:03.591571 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:03.591368    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls podName:2c0e6cc9-602e-4752-92b2-e32a62763c2a nodeName:}" failed. No retries permitted until 2026-09-09 19:56:04.091348622 +0000 UTC m=+36.987370444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls") pod "dns-default-sm8l8" (UID: "2c0e6cc9-602e-4752-92b2-e32a62763c2a") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:03.591788 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.591763    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c0e6cc9-602e-4752-92b2-e32a62763c2a-tmp-dir\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.591935 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.591915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0e6cc9-602e-4752-92b2-e32a62763c2a-config-volume\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.595250 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.595234    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab31a95f-faaf-45ea-9290-95198e492b95-cert\") pod \"ingress-canary-425sq\" (UID: \"ab31a95f-faaf-45ea-9290-95198e492b95\") " pod="openshift-ingress-canary/ingress-canary-425sq"
Sep 09 19:56:03.599538 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.599511    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwlr\" (UniqueName: \"kubernetes.io/projected/2c0e6cc9-602e-4752-92b2-e32a62763c2a-kube-api-access-dtwlr\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:03.599844 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.599825    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxxz\" (UniqueName: \"kubernetes.io/projected/ab31a95f-faaf-45ea-9290-95198e492b95-kube-api-access-xcxxz\") pod \"ingress-canary-425sq\" (UID: \"ab31a95f-faaf-45ea-9290-95198e492b95\") " pod="openshift-ingress-canary/ingress-canary-425sq"
Sep 09 19:56:03.633823 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.633785    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:56:03.636222 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.636204    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:56:03.723082 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.723060    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-425sq"
Sep 09 19:56:03.797953 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.797925    2557 generic.go:356] "Generic (PLEG): container finished" podID="a0d2e226-3707-4893-b20e-1686fddccf6b" containerID="3bd317773ea03309c2b59f746ec65387abec4aed51f76d0a8c60dddf229e126a" exitCode=0
Sep 09 19:56:03.798084 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:03.797994    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59wff" event={"ID":"a0d2e226-3707-4893-b20e-1686fddccf6b","Type":"ContainerDied","Data":"3bd317773ea03309c2b59f746ec65387abec4aed51f76d0a8c60dddf229e126a"}
Sep 09 19:56:04.045816 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.044229    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-425sq"]
Sep 09 19:56:04.048514 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:04.048491    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab31a95f_faaf_45ea_9290_95198e492b95.slice/crio-cbcbcad370b4fb244a5461e4d04c694f9bd987e47b953d3fa3d77b28837a39e4 WatchSource:0}: Error finding container cbcbcad370b4fb244a5461e4d04c694f9bd987e47b953d3fa3d77b28837a39e4: Status 404 returned error can't find the container with id cbcbcad370b4fb244a5461e4d04c694f9bd987e47b953d3fa3d77b28837a39e4
Sep 09 19:56:04.094515 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.094449    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:04.094607 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:04.094590    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:04.094657 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:04.094647    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls podName:2c0e6cc9-602e-4752-92b2-e32a62763c2a nodeName:}" failed. No retries permitted until 2026-09-09 19:56:05.094628398 +0000 UTC m=+37.990650224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls") pod "dns-default-sm8l8" (UID: "2c0e6cc9-602e-4752-92b2-e32a62763c2a") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:04.634016 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.633981    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:56:04.634016 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.634010    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:56:04.636354 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.636316    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:56:04.637094 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.636417    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvgdp\""
Sep 09 19:56:04.637094 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.636570    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:56:04.637094 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.636586    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q8crj\""
Sep 09 19:56:04.637094 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.636664    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:56:04.801132 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.801095    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-425sq" event={"ID":"ab31a95f-faaf-45ea-9290-95198e492b95","Type":"ContainerStarted","Data":"cbcbcad370b4fb244a5461e4d04c694f9bd987e47b953d3fa3d77b28837a39e4"}
Sep 09 19:56:04.804270 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.804241    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-59wff" event={"ID":"a0d2e226-3707-4893-b20e-1686fddccf6b","Type":"ContainerStarted","Data":"c91b92b23f830f473f686341011fc0e9b9c11edc197f34e80ab55e4899d0e479"}
Sep 09 19:56:04.824822 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:04.824785    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-59wff" podStartSLOduration=7.662427536 podStartE2EDuration="36.824771373s" totalImagesPullingTime="29.162343837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:04.823615921 +0000 UTC m=+37.719637746" watchObservedRunningTime="2026-09-09 19:56:04.824771373 +0000 UTC m=+37.720793200"
Sep 09 19:56:05.100421 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:05.100384    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:05.100592 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:05.100515    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:05.100592 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:05.100581    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls podName:2c0e6cc9-602e-4752-92b2-e32a62763c2a nodeName:}" failed. No retries permitted until 2026-09-09 19:56:07.100563199 +0000 UTC m=+39.996585007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls") pod "dns-default-sm8l8" (UID: "2c0e6cc9-602e-4752-92b2-e32a62763c2a") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:07.114625 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:07.114460    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:07.114885 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:07.114604    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:07.114885 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:07.114702    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls podName:2c0e6cc9-602e-4752-92b2-e32a62763c2a nodeName:}" failed. No retries permitted until 2026-09-09 19:56:11.114688447 +0000 UTC m=+44.010710252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls") pod "dns-default-sm8l8" (UID: "2c0e6cc9-602e-4752-92b2-e32a62763c2a") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:07.815203 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:07.815148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-425sq" event={"ID":"ab31a95f-faaf-45ea-9290-95198e492b95","Type":"ContainerStarted","Data":"f2a50a9e1432639290ab8d1862be98ee3f2aec05d7c218209a3c1c6bfa37ceb4"}
Sep 09 19:56:07.830930 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:07.830880    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-425sq" podStartSLOduration=1.781918933 podStartE2EDuration="4.830867571s" totalImagesPullingTime="3.048948638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:07.830392003 +0000 UTC m=+40.726413831" watchObservedRunningTime="2026-09-09 19:56:07.830867571 +0000 UTC m=+40.726889389"
Sep 09 19:56:11.143080 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:11.143039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:11.143500 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:11.143188    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:11.143500 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:11.143247    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls podName:2c0e6cc9-602e-4752-92b2-e32a62763c2a nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.14323047 +0000 UTC m=+52.039252294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls") pod "dns-default-sm8l8" (UID: "2c0e6cc9-602e-4752-92b2-e32a62763c2a") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:13.021560 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.021525    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8"]
Sep 09 19:56:13.052037 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.052006    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8"]
Sep 09 19:56:13.052037 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.052043    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77656c885d-5krh9"]
Sep 09 19:56:13.052228 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.052109    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8"
Sep 09 19:56:13.054561 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.054538    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:56:13.054671 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.054587    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 19:56:13.054671 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.054657    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-wkgtv\""
Sep 09 19:56:13.065920 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.065901    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77656c885d-5krh9"]
Sep 09 19:56:13.066016 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.065977    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.068283 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.068266    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 19:56:13.068283 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.068278    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vz5cf\""
Sep 09 19:56:13.068585 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.068569    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 19:56:13.068706 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.068692    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 19:56:13.078347 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.078326    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 19:56:13.119094 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.119072    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"]
Sep 09 19:56:13.139839 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.139819    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-mglmh"]
Sep 09 19:56:13.139967 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.139949    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.142431 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.142409    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zjxds\""
Sep 09 19:56:13.142542 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.142410    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 19:56:13.142542 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.142410    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:56:13.142628 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.142578    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 19:56:13.142848 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.142830    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 19:56:13.155087 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.155066    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh64l\" (UniqueName: \"kubernetes.io/projected/35446b71-e1a6-47ab-b38e-1daecf7b0ad2-kube-api-access-mh64l\") pod \"volume-data-source-validator-695c895bd6-xxzd8\" (UID: \"35446b71-e1a6-47ab-b38e-1daecf7b0ad2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8"
Sep 09 19:56:13.155191 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.155098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-installation-pull-secrets\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.155191 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.155123    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-image-registry-private-configuration\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.155191 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.155140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-bound-sa-token\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.155319 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.155211    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-certificates\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.155319 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.155232    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6xjf\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-kube-api-access-m6xjf\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.155319 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.155270    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-ca-trust-extracted\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.155319 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.155303    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.155453 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.155340    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-trusted-ca\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.165774 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.165756    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"]
Sep 09 19:56:13.165883 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.165869    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.169327 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.169306    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 19:56:13.169402 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.169325    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 19:56:13.169402 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.169338    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 19:56:13.169780 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.169766    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nhrwk\""
Sep 09 19:56:13.170403 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.170384    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 19:56:13.189708 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.189691    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"]
Sep 09 19:56:13.189842 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.189718    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"]
Sep 09 19:56:13.189842 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.189731    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-mglmh"]
Sep 09 19:56:13.189842 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.189803    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.192354 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.192334    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 19:56:13.192639 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.192625    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 19:56:13.192915 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.192899    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-82bkg\""
Sep 09 19:56:13.192971 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.192918    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 19:56:13.193313 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.193296    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 19:56:13.193700 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.193687    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 19:56:13.233116 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.233095    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"]
Sep 09 19:56:13.253714 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.253697    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"]
Sep 09 19:56:13.253789 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.253772    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.256399 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.256373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-bound-sa-token\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.256496 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.256449    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5697\" (UniqueName: \"kubernetes.io/projected/1506ca70-3614-462f-a385-045d93d6003f-kube-api-access-s5697\") pod \"service-ca-operator-859f94c4b-25pjg\" (UID: \"1506ca70-3614-462f-a385-045d93d6003f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.256595 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.256577    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 19:56:13.256812 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.256796    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 19:56:13.256893 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.256818    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 19:56:13.256893 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.256790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.256893 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.256879    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:56:13.257046 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.256879    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-certificates\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.257046 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257021    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6xjf\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-kube-api-access-m6xjf\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.257149 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-ca-trust-extracted\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.257149 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257136    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-snapshots\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.257269 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qdx\" (UniqueName: \"kubernetes.io/projected/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-kube-api-access-g2qdx\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.257269 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257206    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mzw6h\""
Sep 09 19:56:13.257269 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257242    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1506ca70-3614-462f-a385-045d93d6003f-serving-cert\") pod \"service-ca-operator-859f94c4b-25pjg\" (UID: \"1506ca70-3614-462f-a385-045d93d6003f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.257403 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257311    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-tmp\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.257403 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257361    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.257403 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1506ca70-3614-462f-a385-045d93d6003f-config\") pod \"service-ca-operator-859f94c4b-25pjg\" (UID: \"1506ca70-3614-462f-a385-045d93d6003f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.257567 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257421    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-serving-cert\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.257567 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257452    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-ca-trust-extracted\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.257567 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.257458    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:56:13.257567 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.257489    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77656c885d-5krh9: secret "image-registry-tls" not found
Sep 09 19:56:13.257567 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-trusted-ca\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.257567 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.257539    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls podName:3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b nodeName:}" failed. No retries permitted until 2026-09-09 19:56:13.75751988 +0000 UTC m=+46.653541705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls") pod "image-registry-77656c885d-5krh9" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b") : secret "image-registry-tls" not found
Sep 09 19:56:13.257854 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257570    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh64l\" (UniqueName: \"kubernetes.io/projected/35446b71-e1a6-47ab-b38e-1daecf7b0ad2-kube-api-access-mh64l\") pod \"volume-data-source-validator-695c895bd6-xxzd8\" (UID: \"35446b71-e1a6-47ab-b38e-1daecf7b0ad2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8"
Sep 09 19:56:13.257854 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257600    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.257854 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257602    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-certificates\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.257854 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-installation-pull-secrets\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.257854 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.257694    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-image-registry-private-configuration\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.258244 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.258225    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-trusted-ca\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.261209 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.261190    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-installation-pull-secrets\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.261303 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.261210    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-image-registry-private-configuration\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.273848 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.273774    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-bound-sa-token\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.274386 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.274360    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6xjf\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-kube-api-access-m6xjf\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.275121 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.275094    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh64l\" (UniqueName: \"kubernetes.io/projected/35446b71-e1a6-47ab-b38e-1daecf7b0ad2-kube-api-access-mh64l\") pod \"volume-data-source-validator-695c895bd6-xxzd8\" (UID: \"35446b71-e1a6-47ab-b38e-1daecf7b0ad2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8"
Sep 09 19:56:13.358234 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358207    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qdx\" (UniqueName: \"kubernetes.io/projected/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-kube-api-access-g2qdx\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.358345 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358239    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.358345 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-snapshots\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.358345 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358289    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-tmp\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.358345 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358316    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2aea9318-4fda-49f0-bc0f-39d3c5165401-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tcrdt\" (UID: \"2aea9318-4fda-49f0-bc0f-39d3c5165401\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.358345 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jt9\" (UniqueName: \"kubernetes.io/projected/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-kube-api-access-b4jt9\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.358583 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.358583 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358382    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-serving-cert\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.358583 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358532    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4bqh\" (UniqueName: \"kubernetes.io/projected/2aea9318-4fda-49f0-bc0f-39d3c5165401-kube-api-access-q4bqh\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tcrdt\" (UID: \"2aea9318-4fda-49f0-bc0f-39d3c5165401\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.358738 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358624    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5697\" (UniqueName: \"kubernetes.io/projected/1506ca70-3614-462f-a385-045d93d6003f-kube-api-access-s5697\") pod \"service-ca-operator-859f94c4b-25pjg\" (UID: \"1506ca70-3614-462f-a385-045d93d6003f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.358738 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2aea9318-4fda-49f0-bc0f-39d3c5165401-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tcrdt\" (UID: \"2aea9318-4fda-49f0-bc0f-39d3c5165401\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.358738 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358689    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.358738 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358718    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.358926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358765    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1506ca70-3614-462f-a385-045d93d6003f-serving-cert\") pod \"service-ca-operator-859f94c4b-25pjg\" (UID: \"1506ca70-3614-462f-a385-045d93d6003f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.358926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1506ca70-3614-462f-a385-045d93d6003f-config\") pod \"service-ca-operator-859f94c4b-25pjg\" (UID: \"1506ca70-3614-462f-a385-045d93d6003f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.358926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358854    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-tmp\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.359084 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358933    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-snapshots\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.359084 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.358984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.359277 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.359253    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.359381 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.359366    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1506ca70-3614-462f-a385-045d93d6003f-config\") pod \"service-ca-operator-859f94c4b-25pjg\" (UID: \"1506ca70-3614-462f-a385-045d93d6003f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.360593 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.360576    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8"
Sep 09 19:56:13.360818 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.360798    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-serving-cert\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.365912 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.365894    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qdx\" (UniqueName: \"kubernetes.io/projected/7fa75e95-b8a6-4afe-a78d-f405f9cc3212-kube-api-access-g2qdx\") pod \"insights-operator-76bcd6ccf5-mglmh\" (UID: \"7fa75e95-b8a6-4afe-a78d-f405f9cc3212\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.376931 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.376907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1506ca70-3614-462f-a385-045d93d6003f-serving-cert\") pod \"service-ca-operator-859f94c4b-25pjg\" (UID: \"1506ca70-3614-462f-a385-045d93d6003f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.377123 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.377106    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5697\" (UniqueName: \"kubernetes.io/projected/1506ca70-3614-462f-a385-045d93d6003f-kube-api-access-s5697\") pod \"service-ca-operator-859f94c4b-25pjg\" (UID: \"1506ca70-3614-462f-a385-045d93d6003f\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.448463 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.448436    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"
Sep 09 19:56:13.460057 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.459865    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2aea9318-4fda-49f0-bc0f-39d3c5165401-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tcrdt\" (UID: \"2aea9318-4fda-49f0-bc0f-39d3c5165401\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.460057 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.459899    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jt9\" (UniqueName: \"kubernetes.io/projected/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-kube-api-access-b4jt9\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.460057 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.459917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4bqh\" (UniqueName: \"kubernetes.io/projected/2aea9318-4fda-49f0-bc0f-39d3c5165401-kube-api-access-q4bqh\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tcrdt\" (UID: \"2aea9318-4fda-49f0-bc0f-39d3c5165401\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.460057 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.459943    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2aea9318-4fda-49f0-bc0f-39d3c5165401-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tcrdt\" (UID: \"2aea9318-4fda-49f0-bc0f-39d3c5165401\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.460376 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.460070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.460376 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.460115    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.460376 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.460254    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:13.460376 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.460341    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls podName:9f21e8ad-2bb4-49ec-bc5d-72e1abd32773 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:13.96031283 +0000 UTC m=+46.856334656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxql5" (UID: "9f21e8ad-2bb4-49ec-bc5d-72e1abd32773") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:13.460592 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.460537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2aea9318-4fda-49f0-bc0f-39d3c5165401-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tcrdt\" (UID: \"2aea9318-4fda-49f0-bc0f-39d3c5165401\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.460807 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.460789    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.462146 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.462125    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2aea9318-4fda-49f0-bc0f-39d3c5165401-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tcrdt\" (UID: \"2aea9318-4fda-49f0-bc0f-39d3c5165401\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.467911 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.467892    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jt9\" (UniqueName: \"kubernetes.io/projected/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-kube-api-access-b4jt9\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.468457 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.468441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4bqh\" (UniqueName: \"kubernetes.io/projected/2aea9318-4fda-49f0-bc0f-39d3c5165401-kube-api-access-q4bqh\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tcrdt\" (UID: \"2aea9318-4fda-49f0-bc0f-39d3c5165401\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.474027 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.474009    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh"
Sep 09 19:56:13.569785 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.569758    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"
Sep 09 19:56:13.665746 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.665720    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8"]
Sep 09 19:56:13.668705 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:13.668673    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35446b71_e1a6_47ab_b38e_1daecf7b0ad2.slice/crio-e256a76ab07bcc71dec660efd1f6c142846f58cad1b6d6965eafec4b62c2df29 WatchSource:0}: Error finding container e256a76ab07bcc71dec660efd1f6c142846f58cad1b6d6965eafec4b62c2df29: Status 404 returned error can't find the container with id e256a76ab07bcc71dec660efd1f6c142846f58cad1b6d6965eafec4b62c2df29
Sep 09 19:56:13.735200 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.735151    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg"]
Sep 09 19:56:13.738267 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:13.738245    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1506ca70_3614_462f_a385_045d93d6003f.slice/crio-afe1bdc1909b43bbec48284bc42031b535de54d4008ff05966ccd1dfc102619a WatchSource:0}: Error finding container afe1bdc1909b43bbec48284bc42031b535de54d4008ff05966ccd1dfc102619a: Status 404 returned error can't find the container with id afe1bdc1909b43bbec48284bc42031b535de54d4008ff05966ccd1dfc102619a
Sep 09 19:56:13.759761 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.759739    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-mglmh"]
Sep 09 19:56:13.762307 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:13.762285    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa75e95_b8a6_4afe_a78d_f405f9cc3212.slice/crio-2b85785710834fe2234604119b1ccabc9b3e80a55aa8c89f12ca5b98160b52f8 WatchSource:0}: Error finding container 2b85785710834fe2234604119b1ccabc9b3e80a55aa8c89f12ca5b98160b52f8: Status 404 returned error can't find the container with id 2b85785710834fe2234604119b1ccabc9b3e80a55aa8c89f12ca5b98160b52f8
Sep 09 19:56:13.762381 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.762302    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:13.762478 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.762465    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:56:13.762509 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.762483    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77656c885d-5krh9: secret "image-registry-tls" not found
Sep 09 19:56:13.762547 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.762538    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls podName:3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b nodeName:}" failed. No retries permitted until 2026-09-09 19:56:14.762519933 +0000 UTC m=+47.658541755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls") pod "image-registry-77656c885d-5krh9" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b") : secret "image-registry-tls" not found
Sep 09 19:56:13.832132 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.832061    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh" event={"ID":"7fa75e95-b8a6-4afe-a78d-f405f9cc3212","Type":"ContainerStarted","Data":"2b85785710834fe2234604119b1ccabc9b3e80a55aa8c89f12ca5b98160b52f8"}
Sep 09 19:56:13.833138 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.833114    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg" event={"ID":"1506ca70-3614-462f-a385-045d93d6003f","Type":"ContainerStarted","Data":"afe1bdc1909b43bbec48284bc42031b535de54d4008ff05966ccd1dfc102619a"}
Sep 09 19:56:13.834180 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.834142    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8" event={"ID":"35446b71-e1a6-47ab-b38e-1daecf7b0ad2","Type":"ContainerStarted","Data":"e256a76ab07bcc71dec660efd1f6c142846f58cad1b6d6965eafec4b62c2df29"}
Sep 09 19:56:13.853408 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.853388    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt"]
Sep 09 19:56:13.856353 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:13.856323    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aea9318_4fda_49f0_bc0f_39d3c5165401.slice/crio-07adbd48c1f97c59e727fc395b97e66235e233cea02b6b379f62eb581a9ffcde WatchSource:0}: Error finding container 07adbd48c1f97c59e727fc395b97e66235e233cea02b6b379f62eb581a9ffcde: Status 404 returned error can't find the container with id 07adbd48c1f97c59e727fc395b97e66235e233cea02b6b379f62eb581a9ffcde
Sep 09 19:56:13.964406 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:13.964364    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:13.964573 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.964504    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:13.964573 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:13.964570    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls podName:9f21e8ad-2bb4-49ec-bc5d-72e1abd32773 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:14.964555081 +0000 UTC m=+47.860576887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxql5" (UID: "9f21e8ad-2bb4-49ec-bc5d-72e1abd32773") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:14.772064 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:14.772032    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:14.772508 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:14.772240    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:56:14.772508 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:14.772262    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77656c885d-5krh9: secret "image-registry-tls" not found
Sep 09 19:56:14.772508 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:14.772335    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls podName:3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b nodeName:}" failed. No retries permitted until 2026-09-09 19:56:16.77230868 +0000 UTC m=+49.668330502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls") pod "image-registry-77656c885d-5krh9" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b") : secret "image-registry-tls" not found
Sep 09 19:56:14.838682 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:14.838640    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt" event={"ID":"2aea9318-4fda-49f0-bc0f-39d3c5165401","Type":"ContainerStarted","Data":"07adbd48c1f97c59e727fc395b97e66235e233cea02b6b379f62eb581a9ffcde"}
Sep 09 19:56:14.973278 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:14.973190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:14.973426 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:14.973409    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:14.973511 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:14.973494    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls podName:9f21e8ad-2bb4-49ec-bc5d-72e1abd32773 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:16.97347296 +0000 UTC m=+49.869494782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxql5" (UID: "9f21e8ad-2bb4-49ec-bc5d-72e1abd32773") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:16.790222 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:16.790195    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:16.790544 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:16.790280    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:56:16.790544 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:16.790297    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77656c885d-5krh9: secret "image-registry-tls" not found
Sep 09 19:56:16.790544 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:16.790352    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls podName:3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b nodeName:}" failed. No retries permitted until 2026-09-09 19:56:20.790337201 +0000 UTC m=+53.686359010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls") pod "image-registry-77656c885d-5krh9" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b") : secret "image-registry-tls" not found
Sep 09 19:56:16.992399 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:16.992299    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:16.992580 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:16.992491    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:16.992580 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:16.992562    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls podName:9f21e8ad-2bb4-49ec-bc5d-72e1abd32773 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:20.992540965 +0000 UTC m=+53.888562782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxql5" (UID: "9f21e8ad-2bb4-49ec-bc5d-72e1abd32773") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:17.848991 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:17.848953    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh" event={"ID":"7fa75e95-b8a6-4afe-a78d-f405f9cc3212","Type":"ContainerStarted","Data":"f38a33a5ef258370183ae071c777d8c445cfe5e42f012b4709531ade33f80860"}
Sep 09 19:56:17.851405 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:17.851369    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt" event={"ID":"2aea9318-4fda-49f0-bc0f-39d3c5165401","Type":"ContainerStarted","Data":"af862ffaf0561e5ea80f5ececd3f2358c670847fbecd635e2ae0e39a47ef5562"}
Sep 09 19:56:17.853184 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:17.853141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8" event={"ID":"35446b71-e1a6-47ab-b38e-1daecf7b0ad2","Type":"ContainerStarted","Data":"d474d10865f3da225523346ad36dd5361ced06544210f4185c34ce966139b589"}
Sep 09 19:56:17.866492 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:17.866276    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh" podStartSLOduration=1.878998604 podStartE2EDuration="4.866257413s" totalImagesPullingTime="2.987258809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:17.865515405 +0000 UTC m=+50.761537234" watchObservedRunningTime="2026-09-09 19:56:17.866257413 +0000 UTC m=+50.762279242"
Sep 09 19:56:17.880260 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:17.880208    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-xxzd8" podStartSLOduration=1.854495397 podStartE2EDuration="4.880184761s" totalImagesPullingTime="3.025689364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:17.879391845 +0000 UTC m=+50.775413673" watchObservedRunningTime="2026-09-09 19:56:17.880184761 +0000 UTC m=+50.776206590"
Sep 09 19:56:17.898226 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:17.898025    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt" podStartSLOduration=1.998108929 podStartE2EDuration="4.898003251s" totalImagesPullingTime="2.899894322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:17.896584055 +0000 UTC m=+50.792605883" watchObservedRunningTime="2026-09-09 19:56:17.898003251 +0000 UTC m=+50.794025079"
Sep 09 19:56:18.261945 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.261906    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ms748"]
Sep 09 19:56:18.278623 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.278588    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.280255 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.280232    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ms748"]
Sep 09 19:56:18.280933 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.280913    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 19:56:18.281044 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.280939    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 19:56:18.281044 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.280946    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5xp9v\""
Sep 09 19:56:18.404521 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.404488    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdznr\" (UniqueName: \"kubernetes.io/projected/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-api-access-rdznr\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.404674 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.404531    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-data-volume\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.404674 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.404593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.404674 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.404652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.404786 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.404694    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-crio-socket\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.505514 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.505473    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.505695 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.505546    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.505695 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.505600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-crio-socket\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.505695 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:18.505630    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:18.505695 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:18.505668    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.505695 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:18.505694    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.005677681 +0000 UTC m=+51.901699486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:18.505935 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:18.505732    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:19.005717018 +0000 UTC m=+51.901738863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:18.505935 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.505775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdznr\" (UniqueName: \"kubernetes.io/projected/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-api-access-rdznr\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.505935 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.505792    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-crio-socket\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.505935 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.505814    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-data-volume\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.506112 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.506063    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-data-volume\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.513918 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.513857    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdznr\" (UniqueName: \"kubernetes.io/projected/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-api-access-rdznr\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:18.857345 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.857267    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg" event={"ID":"1506ca70-3614-462f-a385-045d93d6003f","Type":"ContainerStarted","Data":"e6507b83fa1874f60555de992c0fa48caa4cf6894116415a3de415240256c69b"}
Sep 09 19:56:18.873525 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:18.873466    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg" podStartSLOduration=1.01506618 podStartE2EDuration="5.873445578s" totalImagesPullingTime="4.858379398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:18.871657757 +0000 UTC m=+51.767679585" watchObservedRunningTime="2026-09-09 19:56:18.873445578 +0000 UTC m=+51.769467407"
Sep 09 19:56:19.015070 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:19.015030    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:19.015268 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:19.015117    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:19.015268 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:19.015124    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:19.015268 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:19.015212    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:20.015191506 +0000 UTC m=+52.911213331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:19.015550 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:19.015527    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:19.015642 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:19.015599    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:20.015583246 +0000 UTC m=+52.911605079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:19.217975 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:19.217934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:19.218197 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:19.218107    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:19.218252 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:19.218227    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls podName:2c0e6cc9-602e-4752-92b2-e32a62763c2a nodeName:}" failed. No retries permitted until 2026-09-09 19:56:35.218203832 +0000 UTC m=+68.114225669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls") pod "dns-default-sm8l8" (UID: "2c0e6cc9-602e-4752-92b2-e32a62763c2a") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:20.011128 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:20.011096    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9vl9_7d393eee-6f0d-4ca1-bae0-2d669406772c/dns-node-resolver/0.log"
Sep 09 19:56:20.026526 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:20.026493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:20.026659 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:20.026567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:20.026659 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:20.026634    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:20.026743 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:20.026689    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:22.026672523 +0000 UTC m=+54.922694342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:20.026743 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:20.026635    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:20.026852 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:20.026806    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:22.026784061 +0000 UTC m=+54.922805889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:20.833119 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:20.833082    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:20.833303 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:20.833242    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:56:20.833303 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:20.833260    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77656c885d-5krh9: secret "image-registry-tls" not found
Sep 09 19:56:20.833374 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:20.833313    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls podName:3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b nodeName:}" failed. No retries permitted until 2026-09-09 19:56:28.833298362 +0000 UTC m=+61.729320167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls") pod "image-registry-77656c885d-5krh9" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b") : secret "image-registry-tls" not found
Sep 09 19:56:21.011556 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:21.011525    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5d8wp_ec73c477-ff0a-4794-94d2-a905c2c7ba37/node-ca/0.log"
Sep 09 19:56:21.034883 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:21.034849    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:21.035033 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:21.034987    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:21.035072 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:21.035055    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls podName:9f21e8ad-2bb4-49ec-bc5d-72e1abd32773 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:29.035040372 +0000 UTC m=+61.931062177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxql5" (UID: "9f21e8ad-2bb4-49ec-bc5d-72e1abd32773") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:21.810156 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:21.810126    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-425sq_ab31a95f-faaf-45ea-9290-95198e492b95/serve-healthcheck-canary/0.log"
Sep 09 19:56:22.044266 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:22.044222    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:22.044266 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:22.044281    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:22.044688 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:22.044351    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:22.044688 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:22.044385    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:22.044688 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:22.044396    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:26.04438287 +0000 UTC m=+58.940404688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:22.044688 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:22.044468    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:26.044452125 +0000 UTC m=+58.940473934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:26.075467 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:26.075423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:26.075985 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:26.075511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:26.075985 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:26.075585    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:26.075985 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:26.075632    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:26.075985 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:26.075654    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:34.075638037 +0000 UTC m=+66.971659841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:26.075985 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:26.075688    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:34.075674075 +0000 UTC m=+66.971695904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:28.789373 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:28.789346    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smhwd"
Sep 09 19:56:28.898530 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:28.898503    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:28.898702 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:28.898615    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:56:28.898702 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:28.898626    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77656c885d-5krh9: secret "image-registry-tls" not found
Sep 09 19:56:28.898702 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:28.898672    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls podName:3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b nodeName:}" failed. No retries permitted until 2026-09-09 19:56:44.898658159 +0000 UTC m=+77.794679964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls") pod "image-registry-77656c885d-5krh9" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b") : secret "image-registry-tls" not found
Sep 09 19:56:29.100802 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:29.100708    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:29.100942 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:29.100860    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:29.100942 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:29.100938    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls podName:9f21e8ad-2bb4-49ec-bc5d-72e1abd32773 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:45.100922574 +0000 UTC m=+77.996944379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-dxql5" (UID: "9f21e8ad-2bb4-49ec-bc5d-72e1abd32773") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:33.334915 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.334876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:56:33.336991 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.336975    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:56:33.345586 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:33.345559    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 19:56:33.345677 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:33.345635    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs podName:31bdfd97-00d0-4a49-8b1f-8bd09ad7df87 nodeName:}" failed. No retries permitted until 2026-09-09 19:57:37.345613055 +0000 UTC m=+130.241634878 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs") pod "network-metrics-daemon-2mfbb" (UID: "31bdfd97-00d0-4a49-8b1f-8bd09ad7df87") : secret "metrics-daemon-secret" not found
Sep 09 19:56:33.435477 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.435437    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:56:33.437702 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.437681    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:56:33.448589 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.448568    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1fb14a2-9c6e-455e-a59a-bde2e283c026-original-pull-secret\") pod \"global-pull-secret-syncer-6jmkw\" (UID: \"b1fb14a2-9c6e-455e-a59a-bde2e283c026\") " pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:56:33.536879 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.536850    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:56:33.539299 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.539279    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:56:33.549585 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.549562    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:56:33.560175 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.560139    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m4pp\" (UniqueName: \"kubernetes.io/projected/1c4069a3-7ecc-4569-b976-3c59215327b2-kube-api-access-6m4pp\") pod \"network-check-target-dfq9t\" (UID: \"1c4069a3-7ecc-4569-b976-3c59215327b2\") " pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:56:33.642997 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.642931    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6jmkw"
Sep 09 19:56:33.750629 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.750607    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q8crj\""
Sep 09 19:56:33.759274 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.759257    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:56:33.938792 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:33.938764    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6jmkw"]
Sep 09 19:56:33.942341 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:33.942314    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fb14a2_9c6e_455e_a59a_bde2e283c026.slice/crio-41b922eb3f49547f21415501690d6bf636d2b722f385e6a62eacc7683abf1aae WatchSource:0}: Error finding container 41b922eb3f49547f21415501690d6bf636d2b722f385e6a62eacc7683abf1aae: Status 404 returned error can't find the container with id 41b922eb3f49547f21415501690d6bf636d2b722f385e6a62eacc7683abf1aae
Sep 09 19:56:34.049601 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:34.049570    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dfq9t"]
Sep 09 19:56:34.052539 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:34.052511    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4069a3_7ecc_4569_b976_3c59215327b2.slice/crio-aedb05efac0388ec43dc076fec118804f902dd056750ce22d81d0ef12a418c5c WatchSource:0}: Error finding container aedb05efac0388ec43dc076fec118804f902dd056750ce22d81d0ef12a418c5c: Status 404 returned error can't find the container with id aedb05efac0388ec43dc076fec118804f902dd056750ce22d81d0ef12a418c5c
Sep 09 19:56:34.141795 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:34.141759    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:34.141958 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:34.141814    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:34.141958 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:34.141911    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:34.142075 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:34.141980    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls podName:fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:50.141964158 +0000 UTC m=+83.037985963 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ms748" (UID: "fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:34.142393 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:34.142370    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:34.899397 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:34.899326    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6jmkw" event={"ID":"b1fb14a2-9c6e-455e-a59a-bde2e283c026","Type":"ContainerStarted","Data":"41b922eb3f49547f21415501690d6bf636d2b722f385e6a62eacc7683abf1aae"}
Sep 09 19:56:34.900964 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:34.900935    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dfq9t" event={"ID":"1c4069a3-7ecc-4569-b976-3c59215327b2","Type":"ContainerStarted","Data":"aedb05efac0388ec43dc076fec118804f902dd056750ce22d81d0ef12a418c5c"}
Sep 09 19:56:35.250002 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:35.249966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:35.252599 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:35.252567    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c0e6cc9-602e-4752-92b2-e32a62763c2a-metrics-tls\") pod \"dns-default-sm8l8\" (UID: \"2c0e6cc9-602e-4752-92b2-e32a62763c2a\") " pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:35.512722 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:35.512592    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-28vq4\""
Sep 09 19:56:35.521085 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:35.520919    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:35.803193 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:35.803099    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sm8l8"]
Sep 09 19:56:35.807111 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:35.807080    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0e6cc9_602e_4752_92b2_e32a62763c2a.slice/crio-69812f677e13005b9870f91ae8591da571ae0b77a8412ddb93469b1a075d58d6 WatchSource:0}: Error finding container 69812f677e13005b9870f91ae8591da571ae0b77a8412ddb93469b1a075d58d6: Status 404 returned error can't find the container with id 69812f677e13005b9870f91ae8591da571ae0b77a8412ddb93469b1a075d58d6
Sep 09 19:56:35.906672 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:35.906589    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sm8l8" event={"ID":"2c0e6cc9-602e-4752-92b2-e32a62763c2a","Type":"ContainerStarted","Data":"69812f677e13005b9870f91ae8591da571ae0b77a8412ddb93469b1a075d58d6"}
Sep 09 19:56:39.921365 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:39.921329    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6jmkw" event={"ID":"b1fb14a2-9c6e-455e-a59a-bde2e283c026","Type":"ContainerStarted","Data":"23eada62594a93a082736932ad388dcee165ec85f7abdbf7bd02d7dcf7e5a741"}
Sep 09 19:56:39.922824 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:39.922794    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dfq9t" event={"ID":"1c4069a3-7ecc-4569-b976-3c59215327b2","Type":"ContainerStarted","Data":"e902b1d41b02f006c7fd8fc0c97ccdb54d7be7edc5ba0701eea7f61de61587c8"}
Sep 09 19:56:39.922944 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:39.922852    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:56:39.924523 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:39.924494    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sm8l8" event={"ID":"2c0e6cc9-602e-4752-92b2-e32a62763c2a","Type":"ContainerStarted","Data":"668e766c60c4be8c21b43bc30ea6137f57dc2bb09e683a60f82a937178f1d5b7"}
Sep 09 19:56:39.924633 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:39.924555    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sm8l8" event={"ID":"2c0e6cc9-602e-4752-92b2-e32a62763c2a","Type":"ContainerStarted","Data":"4b64b42b78a051531e7cf6b6e73667cf5dd226f28d8979a6204facb8830492f5"}
Sep 09 19:56:39.924633 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:39.924597    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:39.938281 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:39.938225    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6jmkw" podStartSLOduration=66.589761683 podStartE2EDuration="1m11.938206395s" totalImagesPullingTime="5.348444712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:39.938145704 +0000 UTC m=+72.834167533" watchObservedRunningTime="2026-09-09 19:56:39.938206395 +0000 UTC m=+72.834228220"
Sep 09 19:56:39.955336 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:39.955282    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dfq9t" podStartSLOduration=66.703917754 podStartE2EDuration="1m11.955266691s" totalImagesPullingTime="5.251348937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:39.954634275 +0000 UTC m=+72.850656099" watchObservedRunningTime="2026-09-09 19:56:39.955266691 +0000 UTC m=+72.851288517"
Sep 09 19:56:39.972287 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:39.972245    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sm8l8" podStartSLOduration=33.484836647 podStartE2EDuration="36.972214693s" totalImagesPullingTime="3.487378046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:39.971406399 +0000 UTC m=+72.867428232" watchObservedRunningTime="2026-09-09 19:56:39.972214693 +0000 UTC m=+72.868236521"
Sep 09 19:56:44.928238 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:44.928191    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:44.930469 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:44.930449    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"image-registry-77656c885d-5krh9\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") " pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:45.129577 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.129543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:45.132124 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.132092    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f21e8ad-2bb4-49ec-bc5d-72e1abd32773-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-dxql5\" (UID: \"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:45.177295 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.177265    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vz5cf\""
Sep 09 19:56:45.185806 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.185757    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:45.300696 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.300666    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-82bkg\""
Sep 09 19:56:45.309150 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.309129    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"
Sep 09 19:56:45.460965 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.460886    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77656c885d-5krh9"]
Sep 09 19:56:45.463678 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:45.463653    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d152c3a_574c_4ae4_bf82_6b6f8a5cea3b.slice/crio-f300cc87373cbf7280d7fb305f6f3b8e7256f04377714103c88a1ce294205507 WatchSource:0}: Error finding container f300cc87373cbf7280d7fb305f6f3b8e7256f04377714103c88a1ce294205507: Status 404 returned error can't find the container with id f300cc87373cbf7280d7fb305f6f3b8e7256f04377714103c88a1ce294205507
Sep 09 19:56:45.583583 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.583546    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5"]
Sep 09 19:56:45.586187 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:45.586143    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f21e8ad_2bb4_49ec_bc5d_72e1abd32773.slice/crio-c3190cb914fea0bdb30c3fbfad5ea9e4ff97b17fec8499bdd5aa12acf9472dfb WatchSource:0}: Error finding container c3190cb914fea0bdb30c3fbfad5ea9e4ff97b17fec8499bdd5aa12acf9472dfb: Status 404 returned error can't find the container with id c3190cb914fea0bdb30c3fbfad5ea9e4ff97b17fec8499bdd5aa12acf9472dfb
Sep 09 19:56:45.894678 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.894590    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"]
Sep 09 19:56:45.900348 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.900321    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:45.906252 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.906225    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 19:56:45.906493 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.906356    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 19:56:45.906561 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.906493    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 19:56:45.906561 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.906544    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 19:56:45.907330 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.907309    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 19:56:45.907572 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.907556    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 19:56:45.909802 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.909782    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 19:56:45.911227 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.911209    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"]
Sep 09 19:56:45.920695 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.920668    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77656c885d-5krh9"]
Sep 09 19:56:45.937055 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.937029    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:45.937377 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.937063    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-ca\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:45.937377 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.937095    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znmmb\" (UniqueName: \"kubernetes.io/projected/3089a4a0-1026-49d8-9eae-a209e986c924-kube-api-access-znmmb\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:45.937377 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.937140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:45.937377 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.937206    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-hub\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:45.937377 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.937266    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3089a4a0-1026-49d8-9eae-a209e986c924-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:45.943468 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.943431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77656c885d-5krh9" event={"ID":"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b","Type":"ContainerStarted","Data":"f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5"}
Sep 09 19:56:45.943577 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.943498    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:56:45.943577 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.943523    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77656c885d-5krh9" event={"ID":"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b","Type":"ContainerStarted","Data":"f300cc87373cbf7280d7fb305f6f3b8e7256f04377714103c88a1ce294205507"}
Sep 09 19:56:45.944598 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.944577    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5" event={"ID":"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773","Type":"ContainerStarted","Data":"c3190cb914fea0bdb30c3fbfad5ea9e4ff97b17fec8499bdd5aa12acf9472dfb"}
Sep 09 19:56:45.969682 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:45.969625    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77656c885d-5krh9" podStartSLOduration=32.969603785 podStartE2EDuration="32.969603785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:45.968302626 +0000 UTC m=+78.864324453" watchObservedRunningTime="2026-09-09 19:56:45.969603785 +0000 UTC m=+78.865625612"
Sep 09 19:56:46.038033 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.037991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znmmb\" (UniqueName: \"kubernetes.io/projected/3089a4a0-1026-49d8-9eae-a209e986c924-kube-api-access-znmmb\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.038248 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.038033    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.038248 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.038098    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-hub\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.038431 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.038405    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3089a4a0-1026-49d8-9eae-a209e986c924-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.038540 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.038520    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.039444 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.039123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-ca\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.039641 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.039609    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3089a4a0-1026-49d8-9eae-a209e986c924-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.040953 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.040924    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.041088 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.040958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-hub\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.041178 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.041139    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.041585 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.041569    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3089a4a0-1026-49d8-9eae-a209e986c924-ca\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.049023 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.048999    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znmmb\" (UniqueName: \"kubernetes.io/projected/3089a4a0-1026-49d8-9eae-a209e986c924-kube-api-access-znmmb\") pod \"cluster-proxy-proxy-agent-75bb8f86db-bfbcl\" (UID: \"3089a4a0-1026-49d8-9eae-a209e986c924\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.221751 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.221714    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"
Sep 09 19:56:46.495632 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.495541    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl"]
Sep 09 19:56:46.499519 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:46.499494    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3089a4a0_1026_49d8_9eae_a209e986c924.slice/crio-a59c69fdcaeaa541ad9151f46db6842c9ea87325f89082809b38f200e2cf6c92 WatchSource:0}: Error finding container a59c69fdcaeaa541ad9151f46db6842c9ea87325f89082809b38f200e2cf6c92: Status 404 returned error can't find the container with id a59c69fdcaeaa541ad9151f46db6842c9ea87325f89082809b38f200e2cf6c92
Sep 09 19:56:46.949835 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:46.949793    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl" event={"ID":"3089a4a0-1026-49d8-9eae-a209e986c924","Type":"ContainerStarted","Data":"a59c69fdcaeaa541ad9151f46db6842c9ea87325f89082809b38f200e2cf6c92"}
Sep 09 19:56:47.955191 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:47.955120    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5" event={"ID":"9f21e8ad-2bb4-49ec-bc5d-72e1abd32773","Type":"ContainerStarted","Data":"ec66b7a9320edc4b4a1c0b22f3c5e4f5f53eff648d32834765b3cc1f50fce5bf"}
Sep 09 19:56:47.971029 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:47.970974    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-dxql5" podStartSLOduration=32.955394383 podStartE2EDuration="34.970955298s" totalImagesPullingTime="2.015560915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:47.969882375 +0000 UTC m=+80.865904203" watchObservedRunningTime="2026-09-09 19:56:47.970955298 +0000 UTC m=+80.866977116"
Sep 09 19:56:49.929590 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:49.929462    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sm8l8"
Sep 09 19:56:50.174017 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:50.173977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:50.176687 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:50.176660    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ms748\" (UID: \"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9\") " pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:50.392609 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:50.392535    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5xp9v\""
Sep 09 19:56:50.401237 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:50.401201    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ms748"
Sep 09 19:56:50.995330 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:50.995243    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ms748"]
Sep 09 19:56:51.156604 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:51.156572    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0fdb1e_8e87_4c6c_bce4_b4be82ed8ec9.slice/crio-506d45f0e26b82259c2767322b6e4d8e9738deeec96efd7c41e15591bc13b5ce WatchSource:0}: Error finding container 506d45f0e26b82259c2767322b6e4d8e9738deeec96efd7c41e15591bc13b5ce: Status 404 returned error can't find the container with id 506d45f0e26b82259c2767322b6e4d8e9738deeec96efd7c41e15591bc13b5ce
Sep 09 19:56:51.972304 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:51.972260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl" event={"ID":"3089a4a0-1026-49d8-9eae-a209e986c924","Type":"ContainerStarted","Data":"d5c4683b36957f36e26e01cadd1a8eda925fd5159ba0b3f408d6ef0d997fd906"}
Sep 09 19:56:51.972486 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:51.972315    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl" event={"ID":"3089a4a0-1026-49d8-9eae-a209e986c924","Type":"ContainerStarted","Data":"9ff0bdfbe3406d6bfc92b5fc42bfedbb236d622bf9748bbd66d228d4d4ab5bc9"}
Sep 09 19:56:51.972486 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:51.972336    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl" event={"ID":"3089a4a0-1026-49d8-9eae-a209e986c924","Type":"ContainerStarted","Data":"4083617994c97877d1850d321212a374bcfb22927998efed70ba8d27d3de8c33"}
Sep 09 19:56:51.973752 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:51.973729    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ms748" event={"ID":"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9","Type":"ContainerStarted","Data":"e70a2a6864af2af797308e6f321c76ed669df5fd3e87ae8cfa759a2fa81466bd"}
Sep 09 19:56:51.973856 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:51.973783    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ms748" event={"ID":"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9","Type":"ContainerStarted","Data":"506d45f0e26b82259c2767322b6e4d8e9738deeec96efd7c41e15591bc13b5ce"}
Sep 09 19:56:51.994363 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:51.994319    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8f86db-bfbcl" podStartSLOduration=2.298158035 podStartE2EDuration="6.994304342s" totalImagesPullingTime="4.696146307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:51.993140536 +0000 UTC m=+84.889162362" watchObservedRunningTime="2026-09-09 19:56:51.994304342 +0000 UTC m=+84.890326166"
Sep 09 19:56:53.983774 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:53.983731    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ms748" event={"ID":"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9","Type":"ContainerStarted","Data":"de80dc8fa5116ea6e3f04ca7068d2311d0e4a1d62bb11fbe6aa765cfc9363cbc"}
Sep 09 19:56:54.989603 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:54.989567    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ms748" event={"ID":"fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9","Type":"ContainerStarted","Data":"bc599fdf8cd7a23440bebc50acd36925964b1a25d391a4f76cc18b44945d640a"}
Sep 09 19:56:55.012840 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:55.011731    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ms748" podStartSLOduration=33.901772233 podStartE2EDuration="37.01171084s" totalImagesPullingTime="3.109938607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:55.009555264 +0000 UTC m=+87.905577092" watchObservedRunningTime="2026-09-09 19:56:55.01171084 +0000 UTC m=+87.907732670"
Sep 09 19:56:56.632581 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.632543    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7cv9h"]
Sep 09 19:56:56.635910 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.635891    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.638195 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.638171    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 19:56:56.638315 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.638173    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 19:56:56.638315 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.638180    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 19:56:56.638315 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.638181    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 19:56:56.638585 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.638571    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f5wvt\""
Sep 09 19:56:56.727891 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.727860    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d1a993c-544d-4bf8-a879-0ddf547030d6-sys\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.727891 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.727895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-wtmp\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.728123 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.727957    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-tls\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.728123 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.728021    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d1a993c-544d-4bf8-a879-0ddf547030d6-metrics-client-ca\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.728123 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.728045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-textfile\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.728123 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.728073    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.728123 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.728103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.728123 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.728121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d1a993c-544d-4bf8-a879-0ddf547030d6-root\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.728339 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.728184    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zf6w\" (UniqueName: \"kubernetes.io/projected/2d1a993c-544d-4bf8-a879-0ddf547030d6-kube-api-access-4zf6w\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829448 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829417    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d1a993c-544d-4bf8-a879-0ddf547030d6-sys\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829448 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829447    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-wtmp\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829661 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-tls\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829661 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829518    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d1a993c-544d-4bf8-a879-0ddf547030d6-metrics-client-ca\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829661 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829525    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d1a993c-544d-4bf8-a879-0ddf547030d6-sys\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829661 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-textfile\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829661 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829580    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829661 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829606    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-wtmp\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829661 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829625    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.829661 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829657    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d1a993c-544d-4bf8-a879-0ddf547030d6-root\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.830013 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:56.829670    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 19:56:56.830013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829697    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zf6w\" (UniqueName: \"kubernetes.io/projected/2d1a993c-544d-4bf8-a879-0ddf547030d6-kube-api-access-4zf6w\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.830013 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:56:56.829743    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-tls podName:2d1a993c-544d-4bf8-a879-0ddf547030d6 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:57.329719168 +0000 UTC m=+90.225740977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-tls") pod "node-exporter-7cv9h" (UID: "2d1a993c-544d-4bf8-a879-0ddf547030d6") : secret "node-exporter-tls" not found
Sep 09 19:56:56.830013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829806    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d1a993c-544d-4bf8-a879-0ddf547030d6-root\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.830013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.829886    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-textfile\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.830214 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.830197    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d1a993c-544d-4bf8-a879-0ddf547030d6-metrics-client-ca\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.830248 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.830215    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.831670 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.831650    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:56.837791 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:56.837769    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zf6w\" (UniqueName: \"kubernetes.io/projected/2d1a993c-544d-4bf8-a879-0ddf547030d6-kube-api-access-4zf6w\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:57.335594 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:57.335562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-tls\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:57.337818 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:57.337795    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d1a993c-544d-4bf8-a879-0ddf547030d6-node-exporter-tls\") pod \"node-exporter-7cv9h\" (UID: \"2d1a993c-544d-4bf8-a879-0ddf547030d6\") " pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:57.544968 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:57.544935    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7cv9h"
Sep 09 19:56:57.556344 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:56:57.556313    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1a993c_544d_4bf8_a879_0ddf547030d6.slice/crio-da1f422c38b32abbe5faf840fcdbac0d9e6a1561c46659853316afa60071b489 WatchSource:0}: Error finding container da1f422c38b32abbe5faf840fcdbac0d9e6a1561c46659853316afa60071b489: Status 404 returned error can't find the container with id da1f422c38b32abbe5faf840fcdbac0d9e6a1561c46659853316afa60071b489
Sep 09 19:56:58.001455 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:58.001422    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cv9h" event={"ID":"2d1a993c-544d-4bf8-a879-0ddf547030d6","Type":"ContainerStarted","Data":"da1f422c38b32abbe5faf840fcdbac0d9e6a1561c46659853316afa60071b489"}
Sep 09 19:56:59.006116 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:59.006082    2557 generic.go:356] "Generic (PLEG): container finished" podID="2d1a993c-544d-4bf8-a879-0ddf547030d6" containerID="760430168c5d9371393b655ac8d392c2de2735b0cdf64ead228a9cd11b95f8ef" exitCode=0
Sep 09 19:56:59.006677 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:56:59.006123    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cv9h" event={"ID":"2d1a993c-544d-4bf8-a879-0ddf547030d6","Type":"ContainerDied","Data":"760430168c5d9371393b655ac8d392c2de2735b0cdf64ead228a9cd11b95f8ef"}
Sep 09 19:57:00.011913 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:00.011869    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cv9h" event={"ID":"2d1a993c-544d-4bf8-a879-0ddf547030d6","Type":"ContainerStarted","Data":"3639bb68c6d4ff517b9c2f50904f7ad1bc2d03ed5e4c45ff227137fcfcc80e73"}
Sep 09 19:57:00.012339 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:00.011936    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cv9h" event={"ID":"2d1a993c-544d-4bf8-a879-0ddf547030d6","Type":"ContainerStarted","Data":"01dcea75142546ea86d33fbe091f101932e6e7d071709da8aa47b82278a2e799"}
Sep 09 19:57:00.030564 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:00.030511    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7cv9h" podStartSLOduration=3.021377338 podStartE2EDuration="4.030495229s" totalImagesPullingTime="947.858911ms" totalInitContainerRuntime="61.25898ms" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:00.029459158 +0000 UTC m=+92.925480986" watchObservedRunningTime="2026-09-09 19:57:00.030495229 +0000 UTC m=+92.926517056"
Sep 09 19:57:01.347861 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.347828    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"]
Sep 09 19:57:01.350997 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.350981    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:01.352986 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.352954    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8bdh7\""
Sep 09 19:57:01.353113 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.352986    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 19:57:01.359117 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.359096    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"]
Sep 09 19:57:01.468580 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.468548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df4193bb-cba4-44d3-8b1f-247563b1b036-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cwn5h\" (UID: \"df4193bb-cba4-44d3-8b1f-247563b1b036\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:01.468757 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.468590    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gk86\" (UniqueName: \"kubernetes.io/projected/df4193bb-cba4-44d3-8b1f-247563b1b036-kube-api-access-5gk86\") pod \"monitoring-plugin-85bbbbbf4-cwn5h\" (UID: \"df4193bb-cba4-44d3-8b1f-247563b1b036\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:01.570024 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.569974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df4193bb-cba4-44d3-8b1f-247563b1b036-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cwn5h\" (UID: \"df4193bb-cba4-44d3-8b1f-247563b1b036\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:01.570024 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.570025    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gk86\" (UniqueName: \"kubernetes.io/projected/df4193bb-cba4-44d3-8b1f-247563b1b036-kube-api-access-5gk86\") pod \"monitoring-plugin-85bbbbbf4-cwn5h\" (UID: \"df4193bb-cba4-44d3-8b1f-247563b1b036\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:01.570333 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:57:01.570143    2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 19:57:01.570333 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:57:01.570245    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4193bb-cba4-44d3-8b1f-247563b1b036-monitoring-plugin-cert podName:df4193bb-cba4-44d3-8b1f-247563b1b036 nodeName:}" failed. No retries permitted until 2026-09-09 19:57:02.070224713 +0000 UTC m=+94.966246538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/df4193bb-cba4-44d3-8b1f-247563b1b036-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-cwn5h" (UID: "df4193bb-cba4-44d3-8b1f-247563b1b036") : secret "monitoring-plugin-cert" not found
Sep 09 19:57:01.578845 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:01.578820    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gk86\" (UniqueName: \"kubernetes.io/projected/df4193bb-cba4-44d3-8b1f-247563b1b036-kube-api-access-5gk86\") pod \"monitoring-plugin-85bbbbbf4-cwn5h\" (UID: \"df4193bb-cba4-44d3-8b1f-247563b1b036\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:02.074718 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.074675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df4193bb-cba4-44d3-8b1f-247563b1b036-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cwn5h\" (UID: \"df4193bb-cba4-44d3-8b1f-247563b1b036\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:02.077102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.077079    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df4193bb-cba4-44d3-8b1f-247563b1b036-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cwn5h\" (UID: \"df4193bb-cba4-44d3-8b1f-247563b1b036\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:02.260718 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.260682    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:02.532302 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.532271    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"]
Sep 09 19:57:02.535073 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:57:02.535036    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4193bb_cba4_44d3_8b1f_247563b1b036.slice/crio-167df1a3ead2538fbf2d3cf6f4b0d7a1b616a0cedb902c862e6eb574b98464de WatchSource:0}: Error finding container 167df1a3ead2538fbf2d3cf6f4b0d7a1b616a0cedb902c862e6eb574b98464de: Status 404 returned error can't find the container with id 167df1a3ead2538fbf2d3cf6f4b0d7a1b616a0cedb902c862e6eb574b98464de
Sep 09 19:57:02.821698 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.821617    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 19:57:02.826099 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.826081    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.830612 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.830587    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 19:57:02.830612 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.830614    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 19:57:02.830804 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.830620    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 19:57:02.830804 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.830620    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 19:57:02.830985 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.830971    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 19:57:02.831678 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.831530    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 19:57:02.831678 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.831554    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 19:57:02.831678 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.831570    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 19:57:02.831678 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.831583    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 19:57:02.831678 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.831611    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 19:57:02.831922 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.831736    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 19:57:02.831986 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.831920    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 19:57:02.832278 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.832240    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-45bc6ksn9vhs6\""
Sep 09 19:57:02.832278 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.832255    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 19:57:02.832550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.832297    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-74gxb\""
Sep 09 19:57:02.839065 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.839041    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 19:57:02.840983 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.840960    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 19:57:02.842339 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.842320    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 19:57:02.981499 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981460    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981499 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981723 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981723 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981723 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981627    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981723 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981648    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981723 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981723 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981688    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981723 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981756    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981776    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981792    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981816    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lctdq\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-kube-api-access-lctdq\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981866    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981888    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981903    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981943    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.981963 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981964    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:02.982283 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:02.981987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.022572 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.022530    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h" event={"ID":"df4193bb-cba4-44d3-8b1f-247563b1b036","Type":"ContainerStarted","Data":"167df1a3ead2538fbf2d3cf6f4b0d7a1b616a0cedb902c862e6eb574b98464de"}
Sep 09 19:57:03.083433 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083361    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083433 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083433 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083416    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083646 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083446    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083646 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083646 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083496    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083646 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083523    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083646 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083549    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083646 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083646 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.083646 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083655    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lctdq\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-kube-api-access-lctdq\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083722    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083750    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083779    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083807    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083855    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083885    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084013 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.083928    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084488 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.084113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084488 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.084237    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084488 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.084374    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084638 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.084527    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.084638 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.084579    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.085200 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.085156    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.086867 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.086841    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.087375 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.087350    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.087613 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.087583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.087709 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.087632    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.087835 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.087808    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.087920 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.087899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.088036 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.088009    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.088425 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.088385    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.088536 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.088520    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.088866 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.088848    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.089905 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.089882    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.090253 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.090236    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.090312 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.090260    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.094633 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.094613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lctdq\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-kube-api-access-lctdq\") pod \"prometheus-k8s-0\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.138720 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.138688    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:03.421267 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:03.421234    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 19:57:03.424139 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:57:03.424113    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fac894_5e4d_4bda_8adb_0b85c7e2efd2.slice/crio-4ebed3c3f313fb121e04b416e4890f72860d2f80dec0d2941a57956fa5eec495 WatchSource:0}: Error finding container 4ebed3c3f313fb121e04b416e4890f72860d2f80dec0d2941a57956fa5eec495: Status 404 returned error can't find the container with id 4ebed3c3f313fb121e04b416e4890f72860d2f80dec0d2941a57956fa5eec495
Sep 09 19:57:04.028056 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:04.028021    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerStarted","Data":"4ebed3c3f313fb121e04b416e4890f72860d2f80dec0d2941a57956fa5eec495"}
Sep 09 19:57:05.033154 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:05.033065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h" event={"ID":"df4193bb-cba4-44d3-8b1f-247563b1b036","Type":"ContainerStarted","Data":"9175942b6e7705d76ac51021e6e1c847222c2d93aaafeb83acdd1a07cd370230"}
Sep 09 19:57:05.033533 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:05.033241    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:05.037958 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:05.037936    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h"
Sep 09 19:57:05.056295 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:05.056252    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cwn5h" podStartSLOduration=1.8219729390000001 podStartE2EDuration="4.056237625s" totalImagesPullingTime="2.234264686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:05.055945543 +0000 UTC m=+97.951967371" watchObservedRunningTime="2026-09-09 19:57:05.056237625 +0000 UTC m=+97.952259452"
Sep 09 19:57:06.955675 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:06.955646    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:57:07.041782 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:07.041745    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerID="1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192" exitCode=0
Sep 09 19:57:07.041931 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:07.041787    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerDied","Data":"1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192"}
Sep 09 19:57:10.930104 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:10.930075    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dfq9t"
Sep 09 19:57:11.973752 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:11.973713    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77656c885d-5krh9" podUID="3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" containerName="registry" containerID="cri-o://f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5" gracePeriod=30
Sep 09 19:57:12.063587 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.063549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerStarted","Data":"5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213"}
Sep 09 19:57:12.063740 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.063641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerStarted","Data":"c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d"}
Sep 09 19:57:12.264880 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.264851    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:57:12.362394 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.362359    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-ca-trust-extracted\") pod \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") "
Sep 09 19:57:12.362564 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.362409    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-installation-pull-secrets\") pod \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") "
Sep 09 19:57:12.362564 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.362439    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6xjf\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-kube-api-access-m6xjf\") pod \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") "
Sep 09 19:57:12.362564 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.362503    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-bound-sa-token\") pod \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") "
Sep 09 19:57:12.362723 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.362683    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-certificates\") pod \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") "
Sep 09 19:57:12.362784 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.362745    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-image-registry-private-configuration\") pod \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") "
Sep 09 19:57:12.362784 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.362771    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") pod \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") "
Sep 09 19:57:12.362894 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.362814    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-trusted-ca\") pod \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\" (UID: \"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b\") "
Sep 09 19:57:12.363219 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.363188    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:12.363466 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.363444    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:12.365681 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.365656    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:12.365797 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.365776    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:12.365922 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.365897    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:12.366115 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.366089    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-kube-api-access-m6xjf" (OuterVolumeSpecName: "kube-api-access-m6xjf") pod "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b"). InnerVolumeSpecName "kube-api-access-m6xjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:12.366251 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.366229    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:12.372241 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.372217    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" (UID: "3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:57:12.464176 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.464130    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-installation-pull-secrets\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.464176 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.464180    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6xjf\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-kube-api-access-m6xjf\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.464416 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.464197    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-bound-sa-token\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.464416 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.464207    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-certificates\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.464416 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.464218    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-image-registry-private-configuration\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.464416 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.464230    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-registry-tls\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.464416 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.464242    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-trusted-ca\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.464416 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:12.464250    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b-ca-trust-extracted\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:57:13.078274 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.078236    2557 generic.go:356] "Generic (PLEG): container finished" podID="3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" containerID="f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5" exitCode=0
Sep 09 19:57:13.078696 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.078300    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77656c885d-5krh9"
Sep 09 19:57:13.078696 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.078302    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77656c885d-5krh9" event={"ID":"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b","Type":"ContainerDied","Data":"f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5"}
Sep 09 19:57:13.078696 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.078390    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77656c885d-5krh9" event={"ID":"3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b","Type":"ContainerDied","Data":"f300cc87373cbf7280d7fb305f6f3b8e7256f04377714103c88a1ce294205507"}
Sep 09 19:57:13.078696 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.078418    2557 scope.go:117] "RemoveContainer" containerID="f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5"
Sep 09 19:57:13.100590 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.100561    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77656c885d-5krh9"]
Sep 09 19:57:13.105010 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.104985    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77656c885d-5krh9"]
Sep 09 19:57:13.174132 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.174111    2557 scope.go:117] "RemoveContainer" containerID="f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5"
Sep 09 19:57:13.174501 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:57:13.174474    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5\": container with ID starting with f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5 not found: ID does not exist" containerID="f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5"
Sep 09 19:57:13.174612 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.174508    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5"} err="failed to get container status \"f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5\": rpc error: code = NotFound desc = could not find container \"f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5\": container with ID starting with f8c46e387913856a2e4b225af480497c55f37ef03b67a4fa36d60ec62830f2c5 not found: ID does not exist"
Sep 09 19:57:13.638325 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:13.638248    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" path="/var/lib/kubelet/pods/3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b/volumes"
Sep 09 19:57:14.085438 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:14.085403    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerStarted","Data":"06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17"}
Sep 09 19:57:14.085914 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:14.085476    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerStarted","Data":"cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3"}
Sep 09 19:57:14.085914 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:14.085515    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerStarted","Data":"1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001"}
Sep 09 19:57:14.085914 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:14.085554    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerStarted","Data":"981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca"}
Sep 09 19:57:14.114314 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:14.114217    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.01875674 podStartE2EDuration="12.114201243s" totalImagesPullingTime="9.07640822s" totalInitContainerRuntime="19.036283ms" isStatefulPod=false podCreationTimestamp="2026-09-09 19:57:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:14.113376414 +0000 UTC m=+107.009398253" watchObservedRunningTime="2026-09-09 19:57:14.114201243 +0000 UTC m=+107.010223071"
Sep 09 19:57:18.139846 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:18.139804    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:57:37.373545 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:37.373502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:57:37.375874 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:37.375842    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31bdfd97-00d0-4a49-8b1f-8bd09ad7df87-metrics-certs\") pod \"network-metrics-daemon-2mfbb\" (UID: \"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87\") " pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:57:37.646038 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:37.645966    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvgdp\""
Sep 09 19:57:37.654514 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:37.654497    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2mfbb"
Sep 09 19:57:37.928448 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:37.928421    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2mfbb"]
Sep 09 19:57:37.931640 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:57:37.931615    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bdfd97_00d0_4a49_8b1f_8bd09ad7df87.slice/crio-9326c5facaad23da92bf2a5945f50c5d72b39e96584e30a4940e11fa1bedbbb4 WatchSource:0}: Error finding container 9326c5facaad23da92bf2a5945f50c5d72b39e96584e30a4940e11fa1bedbbb4: Status 404 returned error can't find the container with id 9326c5facaad23da92bf2a5945f50c5d72b39e96584e30a4940e11fa1bedbbb4
Sep 09 19:57:38.173020 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:38.172987    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2mfbb" event={"ID":"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87","Type":"ContainerStarted","Data":"9326c5facaad23da92bf2a5945f50c5d72b39e96584e30a4940e11fa1bedbbb4"}
Sep 09 19:57:39.178872 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:39.178834    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2mfbb" event={"ID":"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87","Type":"ContainerStarted","Data":"997ac687bffbe6239c532fe9154a6399a66f3515b3af25d0bb4afe6db44dff6b"}
Sep 09 19:57:39.179261 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:39.178900    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2mfbb" event={"ID":"31bdfd97-00d0-4a49-8b1f-8bd09ad7df87","Type":"ContainerStarted","Data":"1ecfb3b5bbe997c613b499926dfa337328f536006804395994a469fb4a1c56f2"}
Sep 09 19:57:39.197932 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:39.197888    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2mfbb" podStartSLOduration=130.35718918 podStartE2EDuration="2m11.197872265s" totalImagesPullingTime="840.683085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:39.197522888 +0000 UTC m=+132.093544715" watchObservedRunningTime="2026-09-09 19:57:39.197872265 +0000 UTC m=+132.093894114"
Sep 09 19:57:45.200799 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:45.200764    2557 generic.go:356] "Generic (PLEG): container finished" podID="1506ca70-3614-462f-a385-045d93d6003f" containerID="e6507b83fa1874f60555de992c0fa48caa4cf6894116415a3de415240256c69b" exitCode=0
Sep 09 19:57:45.201218 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:45.200836    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg" event={"ID":"1506ca70-3614-462f-a385-045d93d6003f","Type":"ContainerDied","Data":"e6507b83fa1874f60555de992c0fa48caa4cf6894116415a3de415240256c69b"}
Sep 09 19:57:45.201218 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:45.201211    2557 scope.go:117] "RemoveContainer" containerID="e6507b83fa1874f60555de992c0fa48caa4cf6894116415a3de415240256c69b"
Sep 09 19:57:46.206211 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:46.206154    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-25pjg" event={"ID":"1506ca70-3614-462f-a385-045d93d6003f","Type":"ContainerStarted","Data":"eed1419aac97115583bb75b03c877864c91c1a21b4c7a47d9e96027738c88dcc"}
Sep 09 19:57:48.214729 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:48.214691    2557 generic.go:356] "Generic (PLEG): container finished" podID="7fa75e95-b8a6-4afe-a78d-f405f9cc3212" containerID="f38a33a5ef258370183ae071c777d8c445cfe5e42f012b4709531ade33f80860" exitCode=0
Sep 09 19:57:48.215220 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:48.214744    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh" event={"ID":"7fa75e95-b8a6-4afe-a78d-f405f9cc3212","Type":"ContainerDied","Data":"f38a33a5ef258370183ae071c777d8c445cfe5e42f012b4709531ade33f80860"}
Sep 09 19:57:48.215220 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:48.215156    2557 scope.go:117] "RemoveContainer" containerID="f38a33a5ef258370183ae071c777d8c445cfe5e42f012b4709531ade33f80860"
Sep 09 19:57:49.220536 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:49.220493    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-mglmh" event={"ID":"7fa75e95-b8a6-4afe-a78d-f405f9cc3212","Type":"ContainerStarted","Data":"34abe92b59440eb4a518d9d6e016f62e49f9da60bae3ba18981264ddba685fb2"}
Sep 09 19:57:49.221960 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:49.221935    2557 generic.go:356] "Generic (PLEG): container finished" podID="2aea9318-4fda-49f0-bc0f-39d3c5165401" containerID="af862ffaf0561e5ea80f5ececd3f2358c670847fbecd635e2ae0e39a47ef5562" exitCode=0
Sep 09 19:57:49.222061 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:49.221990    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt" event={"ID":"2aea9318-4fda-49f0-bc0f-39d3c5165401","Type":"ContainerDied","Data":"af862ffaf0561e5ea80f5ececd3f2358c670847fbecd635e2ae0e39a47ef5562"}
Sep 09 19:57:49.222271 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:49.222259    2557 scope.go:117] "RemoveContainer" containerID="af862ffaf0561e5ea80f5ececd3f2358c670847fbecd635e2ae0e39a47ef5562"
Sep 09 19:57:50.226914 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:57:50.226880    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tcrdt" event={"ID":"2aea9318-4fda-49f0-bc0f-39d3c5165401","Type":"ContainerStarted","Data":"4f020c79686c45206d4f870f5b14de4f10f4687550ab78400c15f51e1d9bb2cd"}
Sep 09 19:58:03.139802 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:03.139752    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:03.158843 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:03.158812    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:03.287359 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:03.287327    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:21.251349 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:21.251254    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 19:58:21.251796 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:21.251644    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="prometheus" containerID="cri-o://c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d" gracePeriod=600
Sep 09 19:58:21.251796 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:21.251701    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17" gracePeriod=600
Sep 09 19:58:21.251915 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:21.251770    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="thanos-sidecar" containerID="cri-o://981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca" gracePeriod=600
Sep 09 19:58:21.251915 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:21.251763    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy-web" containerID="cri-o://1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001" gracePeriod=600
Sep 09 19:58:21.252012 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:21.251814    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy" containerID="cri-o://cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3" gracePeriod=600
Sep 09 19:58:21.252069 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:21.251789    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="config-reloader" containerID="cri-o://5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213" gracePeriod=600
Sep 09 19:58:22.342213 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342184    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerID="06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17" exitCode=0
Sep 09 19:58:22.342213 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342211    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerID="cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3" exitCode=0
Sep 09 19:58:22.342529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342219    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerID="981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca" exitCode=0
Sep 09 19:58:22.342529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342225    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerID="5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213" exitCode=0
Sep 09 19:58:22.342529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342231    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerID="c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d" exitCode=0
Sep 09 19:58:22.342529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342252    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerDied","Data":"06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17"}
Sep 09 19:58:22.342529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342315    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerDied","Data":"cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3"}
Sep 09 19:58:22.342529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342341    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerDied","Data":"981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca"}
Sep 09 19:58:22.342529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342365    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerDied","Data":"5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213"}
Sep 09 19:58:22.342529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.342389    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerDied","Data":"c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d"}
Sep 09 19:58:22.497947 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.497921    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:22.541852 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.541821    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-web-config\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542019 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.541870    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-thanos-prometheus-http-client-file\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542019 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.541895    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542019 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.541929    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-0\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542019 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.541952    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-1\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542181    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542243 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542218    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-kubelet-serving-ca-bundle\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542361 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542257    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config-out\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542361 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542288    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-metrics-client-certs\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542361 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542325    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542361 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542357    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lctdq\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-kube-api-access-lctdq\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542391    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-kube-rbac-proxy\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542391    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:22.542550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542430    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-2\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542454    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-grpc-tls\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542478    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-db\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542509    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-serving-certs-ca-bundle\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542550 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542533    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-tls\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542880 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542559    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-trusted-ca-bundle\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.542880 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.542834    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.544293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.543190    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:22.544293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.543385    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:22.544293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.543506    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:22.544293 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.544092    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:22.544590 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.544522    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:22.545893 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.545586    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:58:22.545893 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.545862    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:22.546059 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.545975    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:22.546344 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.546311    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config-out" (OuterVolumeSpecName: "config-out") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:58:22.546838 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.546809    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config" (OuterVolumeSpecName: "config") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:22.547384 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.547356    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:22.547976 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.547949    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-kube-api-access-lctdq" (OuterVolumeSpecName: "kube-api-access-lctdq") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "kube-api-access-lctdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:58:22.548224 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.548196    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:22.548337 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.548258    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:22.548397 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.548361    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:22.549789 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.549744    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:22.556980 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.556945    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-web-config" (OuterVolumeSpecName: "web-config") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:58:22.643937 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.643856    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-tls-assets\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.643937 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.643891    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-metrics-client-ca\") pod \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\" (UID: \"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2\") "
Sep 09 19:58:22.644102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644030    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644040    2557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644050    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644059    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-config-out\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644067    2557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-metrics-client-certs\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644077    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644086    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lctdq\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-kube-api-access-lctdq\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644094    2557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-kube-rbac-proxy\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644102 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644103    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644111    2557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-grpc-tls\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644120    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-k8s-db\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644128    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644136    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644145    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644186    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-web-config\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644195    2557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644206    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.644378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.644263    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:58:22.645791 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.645771    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" (UID: "b4fac894-5e4d-4bda-8adb-0b85c7e2efd2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:58:22.744503 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.744471    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-tls-assets\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:22.744503 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:22.744499    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2-configmap-metrics-client-ca\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 19:58:23.349271 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.349237    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerID="1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001" exitCode=0
Sep 09 19:58:23.349626 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.349314    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.349626 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.349311    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerDied","Data":"1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001"}
Sep 09 19:58:23.349626 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.349400    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4fac894-5e4d-4bda-8adb-0b85c7e2efd2","Type":"ContainerDied","Data":"4ebed3c3f313fb121e04b416e4890f72860d2f80dec0d2941a57956fa5eec495"}
Sep 09 19:58:23.349626 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.349450    2557 scope.go:117] "RemoveContainer" containerID="06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17"
Sep 09 19:58:23.359857 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.359836    2557 scope.go:117] "RemoveContainer" containerID="cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3"
Sep 09 19:58:23.366820 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.366804    2557 scope.go:117] "RemoveContainer" containerID="1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001"
Sep 09 19:58:23.373809 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.373785    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 19:58:23.374098 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.374085    2557 scope.go:117] "RemoveContainer" containerID="981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca"
Sep 09 19:58:23.377480 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.377414    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 19:58:23.381906 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.381890    2557 scope.go:117] "RemoveContainer" containerID="5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213"
Sep 09 19:58:23.388290 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.388275    2557 scope.go:117] "RemoveContainer" containerID="c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d"
Sep 09 19:58:23.395111 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.395092    2557 scope.go:117] "RemoveContainer" containerID="1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192"
Sep 09 19:58:23.401434 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.401417    2557 scope.go:117] "RemoveContainer" containerID="06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17"
Sep 09 19:58:23.401717 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:58:23.401700    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17\": container with ID starting with 06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17 not found: ID does not exist" containerID="06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17"
Sep 09 19:58:23.401764 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.401726    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17"} err="failed to get container status \"06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17\": rpc error: code = NotFound desc = could not find container \"06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17\": container with ID starting with 06911420b45dac985a2a83ce28d1daf5ea4560614b11236d2d05285d734a0c17 not found: ID does not exist"
Sep 09 19:58:23.401764 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.401744    2557 scope.go:117] "RemoveContainer" containerID="cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3"
Sep 09 19:58:23.402006 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:58:23.401987    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3\": container with ID starting with cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3 not found: ID does not exist" containerID="cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3"
Sep 09 19:58:23.402055 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.402011    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3"} err="failed to get container status \"cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3\": rpc error: code = NotFound desc = could not find container \"cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3\": container with ID starting with cdf6fa75e3221f161f143f33762bd8fb50ff805743672731decae484381732b3 not found: ID does not exist"
Sep 09 19:58:23.402055 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.402028    2557 scope.go:117] "RemoveContainer" containerID="1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001"
Sep 09 19:58:23.402268 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:58:23.402254    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001\": container with ID starting with 1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001 not found: ID does not exist" containerID="1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001"
Sep 09 19:58:23.402312 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.402270    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001"} err="failed to get container status \"1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001\": rpc error: code = NotFound desc = could not find container \"1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001\": container with ID starting with 1a8e7b055ec5a6cfc2799bf152433befff5b0aea8d42f9f76c541cdd95cff001 not found: ID does not exist"
Sep 09 19:58:23.402312 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.402283    2557 scope.go:117] "RemoveContainer" containerID="981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca"
Sep 09 19:58:23.402543 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:58:23.402524    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca\": container with ID starting with 981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca not found: ID does not exist" containerID="981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca"
Sep 09 19:58:23.402586 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.402547    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca"} err="failed to get container status \"981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca\": rpc error: code = NotFound desc = could not find container \"981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca\": container with ID starting with 981c080859fe68e89c6eb850ea916b58ce4415e1b948cc5722bd436484a9e4ca not found: ID does not exist"
Sep 09 19:58:23.402586 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.402561    2557 scope.go:117] "RemoveContainer" containerID="5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213"
Sep 09 19:58:23.402799 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:58:23.402780    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213\": container with ID starting with 5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213 not found: ID does not exist" containerID="5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213"
Sep 09 19:58:23.402850 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.402808    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213"} err="failed to get container status \"5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213\": rpc error: code = NotFound desc = could not find container \"5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213\": container with ID starting with 5715ebe0728fdc7d06d5ce56bad39a1bb332a5304ae980a70c60b26d7e671213 not found: ID does not exist"
Sep 09 19:58:23.402850 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.402822    2557 scope.go:117] "RemoveContainer" containerID="c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d"
Sep 09 19:58:23.403142 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:58:23.403098    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d\": container with ID starting with c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d not found: ID does not exist" containerID="c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d"
Sep 09 19:58:23.403602 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.403146    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d"} err="failed to get container status \"c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d\": rpc error: code = NotFound desc = could not find container \"c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d\": container with ID starting with c3104ba3655b37545476a0dfe83ef9f11aad9f25ac71dd6f28a16fd244f6ab3d not found: ID does not exist"
Sep 09 19:58:23.403602 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.403590    2557 scope.go:117] "RemoveContainer" containerID="1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192"
Sep 09 19:58:23.403899 ip-10-0-143-121 kubenswrapper[2557]: E0909 19:58:23.403874    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192\": container with ID starting with 1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192 not found: ID does not exist" containerID="1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192"
Sep 09 19:58:23.403962 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.403904    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192"} err="failed to get container status \"1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192\": rpc error: code = NotFound desc = could not find container \"1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192\": container with ID starting with 1b47627696b6bd6e4b31428208065fff9b45d7c94279569a6da37c88dfe49192 not found: ID does not exist"
Sep 09 19:58:23.404668 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404652    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 19:58:23.404887 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404876    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="init-config-reloader"
Sep 09 19:58:23.404926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404889    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="init-config-reloader"
Sep 09 19:58:23.404926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404897    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="thanos-sidecar"
Sep 09 19:58:23.404926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404903    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="thanos-sidecar"
Sep 09 19:58:23.404926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404914    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy-web"
Sep 09 19:58:23.404926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404919    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy-web"
Sep 09 19:58:23.404926 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404925    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy-thanos"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404930    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy-thanos"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404936    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="prometheus"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404941    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="prometheus"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404947    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404953    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404963    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="config-reloader"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404968    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="config-reloader"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404974    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" containerName="registry"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.404980    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" containerName="registry"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.405018    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.405025    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d152c3a-574c-4ae4-bf82-6b6f8a5cea3b" containerName="registry"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.405029    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="config-reloader"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.405034    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy-thanos"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.405038    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="kube-rbac-proxy-web"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.405044    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="prometheus"
Sep 09 19:58:23.405202 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.405049    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" containerName="thanos-sidecar"
Sep 09 19:58:23.408454 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.408437    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.410921 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.410908    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 19:58:23.411008 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.410990    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 19:58:23.411072 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.411033    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-45bc6ksn9vhs6\""
Sep 09 19:58:23.411129 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.411077    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 19:58:23.411196 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.411174    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 19:58:23.411248 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.411190    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 19:58:23.411301 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.411277    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 19:58:23.411529 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.411515    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 19:58:23.412572 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.412543    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 19:58:23.412672 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.412610    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 19:58:23.412672 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.412658    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 19:58:23.413676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.413540    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 19:58:23.413676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.413546    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 19:58:23.413676 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.413597    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-74gxb\""
Sep 09 19:58:23.413840 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.413704    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 19:58:23.419546 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.419517    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 19:58:23.421801 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.421784    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 19:58:23.429123 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.429101    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 19:58:23.449174 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449247    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449279 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449276    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449444 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449293    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449444 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449318    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449444 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ngg\" (UniqueName: \"kubernetes.io/projected/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-kube-api-access-f6ngg\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449444 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449444 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449375    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449444 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449399    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449444 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449425    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-config\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449444 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449439    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449720 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449453    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449720 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449720 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449720 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449527    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449720 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449545    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449720 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.449720 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.449608    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.550458 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550422    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.550627 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550478    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.550627 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550505    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.550627 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550537    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.550627 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550563    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.550843 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.550843 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.550843 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550802    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.550843 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550831    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551031 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551031 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ngg\" (UniqueName: \"kubernetes.io/projected/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-kube-api-access-f6ngg\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551031 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550895    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551031 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550907    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551031 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551031 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.550990    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551031 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.551028    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-config\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551391 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.551051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551391 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.551078    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551391 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.551122    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551391 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.551181    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551391 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.551222    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551391 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.551241    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551674 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.551486    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.551751 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.551730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.552378 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.552031    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.553863 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.553187    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.553863 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.553795    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.553863 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.553818    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.554326 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.554301    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.554689 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.554663    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.555692 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.555198    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.555692 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.555639    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.555692 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.555666    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-config\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.555908 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.555816    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.556064 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.556040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.556302 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.556278    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.556518 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.556493    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.556725 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.556711    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.556859 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.556843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.560712 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.560688    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ngg\" (UniqueName: \"kubernetes.io/projected/81d4b79e-d4f7-4f2c-96f6-21a5070347a0-kube-api-access-f6ngg\") pod \"prometheus-k8s-0\" (UID: \"81d4b79e-d4f7-4f2c-96f6-21a5070347a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:23.638198 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.638116    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4fac894-5e4d-4bda-8adb-0b85c7e2efd2" path="/var/lib/kubelet/pods/b4fac894-5e4d-4bda-8adb-0b85c7e2efd2/volumes"
Sep 09 19:58:23.720524 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:23.720478    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:58:24.005475 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:24.005416    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 19:58:24.007878 ip-10-0-143-121 kubenswrapper[2557]: W0909 19:58:24.007850    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d4b79e_d4f7_4f2c_96f6_21a5070347a0.slice/crio-5b26451bf340564ebb91490d12720437bc67d100a537deab0a3eed2477264874 WatchSource:0}: Error finding container 5b26451bf340564ebb91490d12720437bc67d100a537deab0a3eed2477264874: Status 404 returned error can't find the container with id 5b26451bf340564ebb91490d12720437bc67d100a537deab0a3eed2477264874
Sep 09 19:58:24.355971 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:24.355893    2557 generic.go:356] "Generic (PLEG): container finished" podID="81d4b79e-d4f7-4f2c-96f6-21a5070347a0" containerID="41264c3706e2814bf7df76747c4886c2c9656b9af7c2f31329ed9c206f3fc2a3" exitCode=0
Sep 09 19:58:24.355971 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:24.355966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81d4b79e-d4f7-4f2c-96f6-21a5070347a0","Type":"ContainerDied","Data":"41264c3706e2814bf7df76747c4886c2c9656b9af7c2f31329ed9c206f3fc2a3"}
Sep 09 19:58:24.356362 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:24.356032    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81d4b79e-d4f7-4f2c-96f6-21a5070347a0","Type":"ContainerStarted","Data":"5b26451bf340564ebb91490d12720437bc67d100a537deab0a3eed2477264874"}
Sep 09 19:58:25.364138 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:25.364097    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81d4b79e-d4f7-4f2c-96f6-21a5070347a0","Type":"ContainerStarted","Data":"086ffdcef90e595e92008f695bc03a4b841a1076224db30daa99485e799a3d39"}
Sep 09 19:58:25.364570 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:25.364215    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81d4b79e-d4f7-4f2c-96f6-21a5070347a0","Type":"ContainerStarted","Data":"330a150e011e6ecd7a124dcba86698f74a0f9fbcfc6a2be324e7cbe73a8874e4"}
Sep 09 19:58:25.364570 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:25.364246    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81d4b79e-d4f7-4f2c-96f6-21a5070347a0","Type":"ContainerStarted","Data":"a58d443337207ba99a24d84009c8264f2e894050d57744a3a7ebe7d9997494cf"}
Sep 09 19:58:25.364570 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:25.364278    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81d4b79e-d4f7-4f2c-96f6-21a5070347a0","Type":"ContainerStarted","Data":"b030585c7994a399feef4d4ba0e47924218e5bfcb4e08b26aa7a57f99e3eae7e"}
Sep 09 19:58:25.364570 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:25.364302    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81d4b79e-d4f7-4f2c-96f6-21a5070347a0","Type":"ContainerStarted","Data":"1ac0707348fe1c07f8828fad34515b33197d81d5cf844d70d2a42a38f017bbf2"}
Sep 09 19:58:25.364570 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:25.364343    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81d4b79e-d4f7-4f2c-96f6-21a5070347a0","Type":"ContainerStarted","Data":"c492bb462445e1e1d6acfc1d40033836c606b71db57fff10cc30f08080067b7f"}
Sep 09 19:58:25.391807 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:25.391745    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.373372755 podStartE2EDuration="2.391725745s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.35299ms" isStatefulPod=false podCreationTimestamp="2026-09-09 19:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:58:25.389671874 +0000 UTC m=+178.285693702" watchObservedRunningTime="2026-09-09 19:58:25.391725745 +0000 UTC m=+178.287747574"
Sep 09 19:58:28.720892 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:58:28.720842    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:59:23.720645 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:59:23.720601    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:59:23.735908 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:59:23.735878    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 19:59:24.591898 ip-10-0-143-121 kubenswrapper[2557]: I0909 19:59:24.591873    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:00:27.533192 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:00:27.533146    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:05:19.007513 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.007482    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-x8t7v"]
Sep 09 20:05:19.010583 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.010567    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x8t7v"
Sep 09 20:05:19.012534 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.012515    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 20:05:19.012639 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.012514    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:05:19.012639 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.012519    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hxxjb\""
Sep 09 20:05:19.013013 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.012992    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:05:19.013125 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.012997    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:05:19.018106 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.018085    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x8t7v"]
Sep 09 20:05:19.108830 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.108781    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-kube-api-access-62p9l\") pod \"s3-init-x8t7v\" (UID: \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\") " pod="kserve/s3-init-x8t7v"
Sep 09 20:05:19.109017 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.108859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-model-cache\") pod \"s3-init-x8t7v\" (UID: \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\") " pod="kserve/s3-init-x8t7v"
Sep 09 20:05:19.210091 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.210047    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-model-cache\") pod \"s3-init-x8t7v\" (UID: \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\") " pod="kserve/s3-init-x8t7v"
Sep 09 20:05:19.210276 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.210129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-kube-api-access-62p9l\") pod \"s3-init-x8t7v\" (UID: \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\") " pod="kserve/s3-init-x8t7v"
Sep 09 20:05:19.210447 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.210430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-model-cache\") pod \"s3-init-x8t7v\" (UID: \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\") " pod="kserve/s3-init-x8t7v"
Sep 09 20:05:19.218760 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.218729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-kube-api-access-62p9l\") pod \"s3-init-x8t7v\" (UID: \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\") " pod="kserve/s3-init-x8t7v"
Sep 09 20:05:19.320293 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.320208    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x8t7v"
Sep 09 20:05:19.588283 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.588210    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x8t7v"]
Sep 09 20:05:19.591178 ip-10-0-143-121 kubenswrapper[2557]: W0909 20:05:19.591133    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bb10c96_c99e_4efb_8a1d_4fad7c9fb78f.slice/crio-0e794df45af47098711affaf6d0ae5a99525d8f7dc80490ed6aea781f7f2dd56 WatchSource:0}: Error finding container 0e794df45af47098711affaf6d0ae5a99525d8f7dc80490ed6aea781f7f2dd56: Status 404 returned error can't find the container with id 0e794df45af47098711affaf6d0ae5a99525d8f7dc80490ed6aea781f7f2dd56
Sep 09 20:05:19.593224 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.593204    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:05:19.823515 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:19.823481    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x8t7v" event={"ID":"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f","Type":"ContainerStarted","Data":"0e794df45af47098711affaf6d0ae5a99525d8f7dc80490ed6aea781f7f2dd56"}
Sep 09 20:05:23.842038 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:23.842000    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x8t7v" event={"ID":"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f","Type":"ContainerStarted","Data":"7bb4de5a26aa6aa0783a88d9f64067202e56d5ef4d5bd25c326ceb562776ca7f"}
Sep 09 20:05:24.847093 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:24.847057    2557 generic.go:356] "Generic (PLEG): container finished" podID="9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f" containerID="7bb4de5a26aa6aa0783a88d9f64067202e56d5ef4d5bd25c326ceb562776ca7f" exitCode=0
Sep 09 20:05:24.847453 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:24.847146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x8t7v" event={"ID":"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f","Type":"ContainerDied","Data":"7bb4de5a26aa6aa0783a88d9f64067202e56d5ef4d5bd25c326ceb562776ca7f"}
Sep 09 20:05:29.248922 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:29.248901    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:05:29.871467 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:29.871432    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x8t7v" event={"ID":"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f","Type":"ContainerStarted","Data":"f3a5fdd0831ee5da60d823025f78f2ee55680ac3748023c91ee43e7a24a5f0a3"}
Sep 09 20:05:29.888306 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:05:29.888255    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-x8t7v" podStartSLOduration=2.385826005 podStartE2EDuration="11.888239363s" totalImagesPullingTime="7.801068082s" totalInitContainerRuntime="1.701345276s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:05:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:05:29.887574654 +0000 UTC m=+602.783596480" watchObservedRunningTime="2026-09-09 20:05:29.888239363 +0000 UTC m=+602.784261190"
Sep 09 20:06:35.109304 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:35.109264    2557 generic.go:356] "Generic (PLEG): container finished" podID="9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f" containerID="f3a5fdd0831ee5da60d823025f78f2ee55680ac3748023c91ee43e7a24a5f0a3" exitCode=0
Sep 09 20:06:35.109724 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:35.109338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x8t7v" event={"ID":"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f","Type":"ContainerDied","Data":"f3a5fdd0831ee5da60d823025f78f2ee55680ac3748023c91ee43e7a24a5f0a3"}
Sep 09 20:06:36.235707 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:36.235684    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x8t7v"
Sep 09 20:06:36.343080 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:36.343050    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-model-cache\") pod \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\" (UID: \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\") "
Sep 09 20:06:36.343269 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:36.343117    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-kube-api-access-62p9l\") pod \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\" (UID: \"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f\") "
Sep 09 20:06:36.345245 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:36.345208    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-kube-api-access-62p9l" (OuterVolumeSpecName: "kube-api-access-62p9l") pod "9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f" (UID: "9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f"). InnerVolumeSpecName "kube-api-access-62p9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:06:36.361997 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:36.361973    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f" (UID: "9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:06:36.443942 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:36.443906    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-kube-api-access-62p9l\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:06:36.443942 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:36.443933    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f-model-cache\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:06:37.117824 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:37.117792    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x8t7v"
Sep 09 20:06:37.118014 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:37.117798    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x8t7v" event={"ID":"9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f","Type":"ContainerDied","Data":"0e794df45af47098711affaf6d0ae5a99525d8f7dc80490ed6aea781f7f2dd56"}
Sep 09 20:06:37.118014 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:37.117903    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e794df45af47098711affaf6d0ae5a99525d8f7dc80490ed6aea781f7f2dd56"
Sep 09 20:06:52.029413 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.029374    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"]
Sep 09 20:06:52.029865 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.029690    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f" containerName="download-hf-model"
Sep 09 20:06:52.029865 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.029702    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f" containerName="download-hf-model"
Sep 09 20:06:52.029865 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.029710    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f" containerName="s3-init"
Sep 09 20:06:52.029865 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.029716    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f" containerName="s3-init"
Sep 09 20:06:52.029865 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.029759    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f" containerName="s3-init"
Sep 09 20:06:52.032628 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.032610    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.034774 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.034748    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:06:52.034909 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.034817    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:06:52.034909 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.034836    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:06:52.034909 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.034868    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:06:52.035079 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.034986    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:06:52.035216 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.035199    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:06:52.035318 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.035238    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:06:52.045657 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.045635    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"]
Sep 09 20:06:52.066820 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.066787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded11425-d936-494b-bdb3-6432f1d8df09-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.066975 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.066832    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.066975 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.066852    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.066975 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.066959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.067129 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.066993    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2db69\" (UniqueName: \"kubernetes.io/projected/ded11425-d936-494b-bdb3-6432f1d8df09-kube-api-access-2db69\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.168131 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.168092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded11425-d936-494b-bdb3-6432f1d8df09-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.168131 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.168140    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.168408 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.168176    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.168408 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.168203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.168408 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.168218    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2db69\" (UniqueName: \"kubernetes.io/projected/ded11425-d936-494b-bdb3-6432f1d8df09-kube-api-access-2db69\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.168408 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:06:52.168282    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:06:52.168408 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:06:52.168347    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls podName:ded11425-d936-494b-bdb3-6432f1d8df09 nodeName:}" failed. No retries permitted until 2026-09-09 20:06:52.668327692 +0000 UTC m=+685.564349503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls") pod "isvc-canary-rollback-predictor-585bd9bcc8-l8852" (UID: "ded11425-d936-494b-bdb3-6432f1d8df09") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:06:52.168674 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.168566    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded11425-d936-494b-bdb3-6432f1d8df09-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.168846 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.168827    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.168937 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.168913    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.178669 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.178643    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2db69\" (UniqueName: \"kubernetes.io/projected/ded11425-d936-494b-bdb3-6432f1d8df09-kube-api-access-2db69\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.672388 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:52.672347    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:52.672595 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:06:52.672528    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:06:52.672664 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:06:52.672622    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls podName:ded11425-d936-494b-bdb3-6432f1d8df09 nodeName:}" failed. No retries permitted until 2026-09-09 20:06:53.672603419 +0000 UTC m=+686.568625230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls") pod "isvc-canary-rollback-predictor-585bd9bcc8-l8852" (UID: "ded11425-d936-494b-bdb3-6432f1d8df09") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:06:53.683400 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:53.683363    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:53.685720 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:53.685700    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-l8852\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:53.843966 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:53.843928    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:06:54.117964 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:54.117800    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"]
Sep 09 20:06:54.120758 ip-10-0-143-121 kubenswrapper[2557]: W0909 20:06:54.120722    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded11425_d936_494b_bdb3_6432f1d8df09.slice/crio-70117d6896629cb23d644712d517a00874ab3f2651904827e6357c205614e233 WatchSource:0}: Error finding container 70117d6896629cb23d644712d517a00874ab3f2651904827e6357c205614e233: Status 404 returned error can't find the container with id 70117d6896629cb23d644712d517a00874ab3f2651904827e6357c205614e233
Sep 09 20:06:54.182111 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:54.182079    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" event={"ID":"ded11425-d936-494b-bdb3-6432f1d8df09","Type":"ContainerStarted","Data":"3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c"}
Sep 09 20:06:54.182250 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:54.182151    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" event={"ID":"ded11425-d936-494b-bdb3-6432f1d8df09","Type":"ContainerStarted","Data":"70117d6896629cb23d644712d517a00874ab3f2651904827e6357c205614e233"}
Sep 09 20:06:58.198771 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:58.198739    2557 generic.go:356] "Generic (PLEG): container finished" podID="ded11425-d936-494b-bdb3-6432f1d8df09" containerID="3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c" exitCode=0
Sep 09 20:06:58.199269 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:06:58.198818    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" event={"ID":"ded11425-d936-494b-bdb3-6432f1d8df09","Type":"ContainerDied","Data":"3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c"}
Sep 09 20:07:13.274506 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:13.274471    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" event={"ID":"ded11425-d936-494b-bdb3-6432f1d8df09","Type":"ContainerStarted","Data":"bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543"}
Sep 09 20:07:15.284825 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:15.284796    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" event={"ID":"ded11425-d936-494b-bdb3-6432f1d8df09","Type":"ContainerStarted","Data":"61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f"}
Sep 09 20:07:15.285262 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:15.284903    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:07:15.308628 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:15.308567    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podStartSLOduration=2.7578404499999998 podStartE2EDuration="23.308553302s" totalImagesPullingTime="16.91546613s" totalInitContainerRuntime="3.635246722s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:07:15.307965436 +0000 UTC m=+708.203987263" watchObservedRunningTime="2026-09-09 20:07:15.308553302 +0000 UTC m=+708.204575128"
Sep 09 20:07:16.288538 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:16.288503    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:07:16.289760 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:16.289727    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:07:17.293058 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:17.293013    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:07:22.297724 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:22.297651    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:07:22.298208 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:22.298183    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:07:32.298908 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:32.298863    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:07:42.299109 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:42.299069    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:07:52.298235 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:07:52.298199    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:08:02.298473 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:08:02.298431    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:08:12.298593 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:08:12.298549    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:08:22.298772 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:08:22.298732    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:08:32.298785 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:08:32.298757    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:10:11.862796 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:11.862764    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"]
Sep 09 20:10:11.863337 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:11.863077    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" containerID="cri-o://bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543" gracePeriod=30
Sep 09 20:10:11.863337 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:11.863119    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kube-rbac-proxy" containerID="cri-o://61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f" gracePeriod=30
Sep 09 20:10:12.293780 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:12.293731    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 09 20:10:12.299147 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:12.299122    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 09 20:10:12.939041 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:12.939011    2557 generic.go:356] "Generic (PLEG): container finished" podID="ded11425-d936-494b-bdb3-6432f1d8df09" containerID="61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f" exitCode=2
Sep 09 20:10:12.939422 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:12.939057    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" event={"ID":"ded11425-d936-494b-bdb3-6432f1d8df09","Type":"ContainerDied","Data":"61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f"}
Sep 09 20:10:16.497657 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.497635    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:10:16.585801 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.585766    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded11425-d936-494b-bdb3-6432f1d8df09-kserve-provision-location\") pod \"ded11425-d936-494b-bdb3-6432f1d8df09\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") "
Sep 09 20:10:16.585966 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.585811    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-cabundle-cert\") pod \"ded11425-d936-494b-bdb3-6432f1d8df09\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") "
Sep 09 20:10:16.585966 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.585833    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls\") pod \"ded11425-d936-494b-bdb3-6432f1d8df09\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") "
Sep 09 20:10:16.585966 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.585854    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2db69\" (UniqueName: \"kubernetes.io/projected/ded11425-d936-494b-bdb3-6432f1d8df09-kube-api-access-2db69\") pod \"ded11425-d936-494b-bdb3-6432f1d8df09\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") "
Sep 09 20:10:16.585966 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.585921    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-kube-rbac-proxy-sar-config\") pod \"ded11425-d936-494b-bdb3-6432f1d8df09\" (UID: \"ded11425-d936-494b-bdb3-6432f1d8df09\") "
Sep 09 20:10:16.586234 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.586201    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded11425-d936-494b-bdb3-6432f1d8df09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ded11425-d936-494b-bdb3-6432f1d8df09" (UID: "ded11425-d936-494b-bdb3-6432f1d8df09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:10:16.586332 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.586232    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ded11425-d936-494b-bdb3-6432f1d8df09" (UID: "ded11425-d936-494b-bdb3-6432f1d8df09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:16.586393 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.586357    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ded11425-d936-494b-bdb3-6432f1d8df09" (UID: "ded11425-d936-494b-bdb3-6432f1d8df09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:16.588065 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.588035    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ded11425-d936-494b-bdb3-6432f1d8df09" (UID: "ded11425-d936-494b-bdb3-6432f1d8df09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:10:16.588157 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.588086    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded11425-d936-494b-bdb3-6432f1d8df09-kube-api-access-2db69" (OuterVolumeSpecName: "kube-api-access-2db69") pod "ded11425-d936-494b-bdb3-6432f1d8df09" (UID: "ded11425-d936-494b-bdb3-6432f1d8df09"). InnerVolumeSpecName "kube-api-access-2db69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:10:16.687119 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.687084    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-cabundle-cert\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:10:16.687119 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.687116    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded11425-d936-494b-bdb3-6432f1d8df09-proxy-tls\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:10:16.687342 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.687129    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2db69\" (UniqueName: \"kubernetes.io/projected/ded11425-d936-494b-bdb3-6432f1d8df09-kube-api-access-2db69\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:10:16.687342 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.687138    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded11425-d936-494b-bdb3-6432f1d8df09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:10:16.687342 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.687147    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded11425-d936-494b-bdb3-6432f1d8df09-kserve-provision-location\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:10:16.957619 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.957529    2557 generic.go:356] "Generic (PLEG): container finished" podID="ded11425-d936-494b-bdb3-6432f1d8df09" containerID="bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543" exitCode=0
Sep 09 20:10:16.957619 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.957586    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"
Sep 09 20:10:16.957792 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.957614    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" event={"ID":"ded11425-d936-494b-bdb3-6432f1d8df09","Type":"ContainerDied","Data":"bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543"}
Sep 09 20:10:16.957792 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.957670    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852" event={"ID":"ded11425-d936-494b-bdb3-6432f1d8df09","Type":"ContainerDied","Data":"70117d6896629cb23d644712d517a00874ab3f2651904827e6357c205614e233"}
Sep 09 20:10:16.957792 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.957699    2557 scope.go:117] "RemoveContainer" containerID="61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f"
Sep 09 20:10:16.965921 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.965906    2557 scope.go:117] "RemoveContainer" containerID="bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543"
Sep 09 20:10:16.972902 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.972884    2557 scope.go:117] "RemoveContainer" containerID="3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c"
Sep 09 20:10:16.977988 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.977961    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"]
Sep 09 20:10:16.980099 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.980078    2557 scope.go:117] "RemoveContainer" containerID="61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f"
Sep 09 20:10:16.980386 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:10:16.980370    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f\": container with ID starting with 61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f not found: ID does not exist" containerID="61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f"
Sep 09 20:10:16.980455 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.980393    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f"} err="failed to get container status \"61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f\": rpc error: code = NotFound desc = could not find container \"61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f\": container with ID starting with 61a224c05acd3def501706cf012f419225b1d36bb1b2341c32be77390b97d91f not found: ID does not exist"
Sep 09 20:10:16.980455 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.980409    2557 scope.go:117] "RemoveContainer" containerID="bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543"
Sep 09 20:10:16.980659 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:10:16.980641    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543\": container with ID starting with bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543 not found: ID does not exist" containerID="bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543"
Sep 09 20:10:16.980721 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.980669    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543"} err="failed to get container status \"bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543\": rpc error: code = NotFound desc = could not find container \"bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543\": container with ID starting with bc396972f303f54724a715441f84b586630088b327893d0fdfa0449b8120e543 not found: ID does not exist"
Sep 09 20:10:16.980721 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.980694    2557 scope.go:117] "RemoveContainer" containerID="3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c"
Sep 09 20:10:16.980931 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:10:16.980915    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c\": container with ID starting with 3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c not found: ID does not exist" containerID="3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c"
Sep 09 20:10:16.980983 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.980938    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c"} err="failed to get container status \"3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c\": rpc error: code = NotFound desc = could not find container \"3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c\": container with ID starting with 3ddd2702e8ffc9d4c4fe84423b2d673cb0d0fdd4eff01ce7ad482d9828ede95c not found: ID does not exist"
Sep 09 20:10:16.986074 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:16.986049    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-l8852"]
Sep 09 20:10:17.108177 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.108133    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"]
Sep 09 20:10:17.108412 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.108399    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container"
Sep 09 20:10:17.108461 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.108414    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container"
Sep 09 20:10:17.108461 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.108439    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="storage-initializer"
Sep 09 20:10:17.108461 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.108444    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="storage-initializer"
Sep 09 20:10:17.108461 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.108454    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kube-rbac-proxy"
Sep 09 20:10:17.108461 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.108459    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kube-rbac-proxy"
Sep 09 20:10:17.108604 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.108498    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kube-rbac-proxy"
Sep 09 20:10:17.108604 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.108506    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" containerName="kserve-container"
Sep 09 20:10:17.112131 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.112110    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.114039 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.114021    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:10:17.114512 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.114480    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:10:17.114610 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.114487    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 20:10:17.114731 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.114718    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:10:17.114978 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.114964    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:10:17.119273 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.115478    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 20:10:17.120360 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.120334    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fh9bz\""
Sep 09 20:10:17.125552 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.125527    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"]
Sep 09 20:10:17.190398 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.190360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.190398 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.190396    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.190607 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.190428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.190607 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.190506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8phzh\" (UniqueName: \"kubernetes.io/projected/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-api-access-8phzh\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.190607 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.190536    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.291371 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.291280    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8phzh\" (UniqueName: \"kubernetes.io/projected/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-api-access-8phzh\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.291371 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.291326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.291562 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.291389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.291562 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.291413    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.291562 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.291455    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.291802 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.291779    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.292182 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.292146    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.292258 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.292151    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.293802 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.293783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.301250 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.301229    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8phzh\" (UniqueName: \"kubernetes.io/projected/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-api-access-8phzh\") pod \"isvc-canary-stop-predictor-5d5b899946-kdt22\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.425695 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.425657    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:17.637971 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.637890    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded11425-d936-494b-bdb3-6432f1d8df09" path="/var/lib/kubelet/pods/ded11425-d936-494b-bdb3-6432f1d8df09/volumes"
Sep 09 20:10:17.697917 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.697837    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"]
Sep 09 20:10:17.700524 ip-10-0-143-121 kubenswrapper[2557]: W0909 20:10:17.700498    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e6cb05_b02c_445a_8b0c_210f0dc29abf.slice/crio-c387a8e3e6fbc9ec615c0ceb090a56b8a1650d2207e52ffe29cd4dbf9dfc0e1d WatchSource:0}: Error finding container c387a8e3e6fbc9ec615c0ceb090a56b8a1650d2207e52ffe29cd4dbf9dfc0e1d: Status 404 returned error can't find the container with id c387a8e3e6fbc9ec615c0ceb090a56b8a1650d2207e52ffe29cd4dbf9dfc0e1d
Sep 09 20:10:17.964579 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.964542    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" event={"ID":"a3e6cb05-b02c-445a-8b0c-210f0dc29abf","Type":"ContainerStarted","Data":"ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c"}
Sep 09 20:10:17.964757 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:17.964605    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" event={"ID":"a3e6cb05-b02c-445a-8b0c-210f0dc29abf","Type":"ContainerStarted","Data":"c387a8e3e6fbc9ec615c0ceb090a56b8a1650d2207e52ffe29cd4dbf9dfc0e1d"}
Sep 09 20:10:21.981801 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:21.981769    2557 generic.go:356] "Generic (PLEG): container finished" podID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerID="ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c" exitCode=0
Sep 09 20:10:21.982290 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:21.981849    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" event={"ID":"a3e6cb05-b02c-445a-8b0c-210f0dc29abf","Type":"ContainerDied","Data":"ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c"}
Sep 09 20:10:22.987177 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:22.987134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" event={"ID":"a3e6cb05-b02c-445a-8b0c-210f0dc29abf","Type":"ContainerStarted","Data":"4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a"}
Sep 09 20:10:22.987534 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:22.987209    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" event={"ID":"a3e6cb05-b02c-445a-8b0c-210f0dc29abf","Type":"ContainerStarted","Data":"d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a"}
Sep 09 20:10:22.987534 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:22.987392    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:23.006108 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:23.006057    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podStartSLOduration=2.292212578 podStartE2EDuration="6.006039436s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.713826858s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:23.005348339 +0000 UTC m=+895.901370167" watchObservedRunningTime="2026-09-09 20:10:23.006039436 +0000 UTC m=+895.902061263"
Sep 09 20:10:23.991598 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:23.991562    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:23.992927 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:23.992899    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:10:24.996232 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:24.996177    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:10:30.000515 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:30.000487    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:10:30.001014 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:30.000987    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:10:37.408029 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.407992    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"]
Sep 09 20:10:37.411022 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.411004    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.414715 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.414689    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-5164f-predictor-serving-cert\""
Sep 09 20:10:37.415453 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.415435    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-5164f-kube-rbac-proxy-sar-config\""
Sep 09 20:10:37.422877 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.422851    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"]
Sep 09 20:10:37.565981 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.565942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.565981 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.565982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5adce253-03bf-4f36-97c2-4e0b0be3b257-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.566238 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.566013    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm5vh\" (UniqueName: \"kubernetes.io/projected/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-api-access-tm5vh\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.566238 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.566099    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.566238 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.566145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5adce253-03bf-4f36-97c2-4e0b0be3b257-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.667234 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.667135    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm5vh\" (UniqueName: \"kubernetes.io/projected/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-api-access-tm5vh\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.667396 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.667238    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.667396 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.667285    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5adce253-03bf-4f36-97c2-4e0b0be3b257-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.667396 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.667325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.667396 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.667351    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5adce253-03bf-4f36-97c2-4e0b0be3b257-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.667588 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:10:37.667463    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-5164f-predictor-serving-cert" not found
Sep 09 20:10:37.667588 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:10:37.667537    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5adce253-03bf-4f36-97c2-4e0b0be3b257-proxy-tls podName:5adce253-03bf-4f36-97c2-4e0b0be3b257 nodeName:}" failed. No retries permitted until 2026-09-09 20:10:38.167516873 +0000 UTC m=+911.063538679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5adce253-03bf-4f36-97c2-4e0b0be3b257-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" (UID: "5adce253-03bf-4f36-97c2-4e0b0be3b257") : secret "isvc-xgboost-graph-raw-hpa-5164f-predictor-serving-cert" not found
Sep 09 20:10:37.667725 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.667706    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5adce253-03bf-4f36-97c2-4e0b0be3b257-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.667941 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.667923    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.667997 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.667980    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:37.675811 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:37.675785    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm5vh\" (UniqueName: \"kubernetes.io/projected/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-api-access-tm5vh\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:38.172099 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:38.172065    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5adce253-03bf-4f36-97c2-4e0b0be3b257-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:38.174497 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:38.174474    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5adce253-03bf-4f36-97c2-4e0b0be3b257-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:38.321284 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:38.321248    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:10:38.596195 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:38.596148    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"]
Sep 09 20:10:38.598769 ip-10-0-143-121 kubenswrapper[2557]: W0909 20:10:38.598741    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5adce253_03bf_4f36_97c2_4e0b0be3b257.slice/crio-03e60ddcacadf0579d65ccf31368d4bd51de4e3f94d37f9fdda98f808e3166bb WatchSource:0}: Error finding container 03e60ddcacadf0579d65ccf31368d4bd51de4e3f94d37f9fdda98f808e3166bb: Status 404 returned error can't find the container with id 03e60ddcacadf0579d65ccf31368d4bd51de4e3f94d37f9fdda98f808e3166bb
Sep 09 20:10:39.053458 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:39.053420    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" event={"ID":"5adce253-03bf-4f36-97c2-4e0b0be3b257","Type":"ContainerStarted","Data":"1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4"}
Sep 09 20:10:39.053629 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:39.053490    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" event={"ID":"5adce253-03bf-4f36-97c2-4e0b0be3b257","Type":"ContainerStarted","Data":"03e60ddcacadf0579d65ccf31368d4bd51de4e3f94d37f9fdda98f808e3166bb"}
Sep 09 20:10:40.001722 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:40.001681    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:10:43.069712 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:43.069678    2557 generic.go:356] "Generic (PLEG): container finished" podID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerID="1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4" exitCode=0
Sep 09 20:10:43.070202 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:43.069751    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" event={"ID":"5adce253-03bf-4f36-97c2-4e0b0be3b257","Type":"ContainerDied","Data":"1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4"}
Sep 09 20:10:43.071036 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:43.071023    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:10:50.001806 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:10:50.001753    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:11:00.001264 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:00.001217    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:11:03.156768 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:03.156728    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" event={"ID":"5adce253-03bf-4f36-97c2-4e0b0be3b257","Type":"ContainerStarted","Data":"369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf"}
Sep 09 20:11:03.157270 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:03.156803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" event={"ID":"5adce253-03bf-4f36-97c2-4e0b0be3b257","Type":"ContainerStarted","Data":"e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4"}
Sep 09 20:11:03.157270 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:03.157018    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:11:03.176900 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:03.176847    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podStartSLOduration=2.906686041 podStartE2EDuration="26.176809742s" totalImagesPullingTime="19.636275683s" totalInitContainerRuntime="3.633848018s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:03.176414042 +0000 UTC m=+936.072435853" watchObservedRunningTime="2026-09-09 20:11:03.176809742 +0000 UTC m=+936.072831570"
Sep 09 20:11:04.160817 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:04.160781    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:11:04.162233 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:04.162200    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 20:11:05.164265 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:05.164224    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 20:11:10.001394 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:10.001354    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:11:10.168736 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:10.168706    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:11:10.169244 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:10.169214    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 20:11:20.001009 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:20.000970    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:11:20.169218 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:20.169151    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 20:11:30.001386 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:30.001345    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:11:30.170238 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:30.170199    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 20:11:40.001349 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:40.001317    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:11:40.169946 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:40.169908    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 20:11:46.671911 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:46.671872    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"]
Sep 09 20:11:46.672408 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:46.672106    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" containerID="cri-o://d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a" gracePeriod=30
Sep 09 20:11:46.672408 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:46.672181    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kube-rbac-proxy" containerID="cri-o://4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a" gracePeriod=30
Sep 09 20:11:47.330254 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:47.330220    2557 generic.go:356] "Generic (PLEG): container finished" podID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerID="4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a" exitCode=2
Sep 09 20:11:47.330430 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:47.330293    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" event={"ID":"a3e6cb05-b02c-445a-8b0c-210f0dc29abf","Type":"ContainerDied","Data":"4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a"}
Sep 09 20:11:49.996484 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:49.996400    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 09 20:11:50.000971 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:50.000933    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 09 20:11:50.169186 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:50.169126    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 20:11:51.907307 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:51.907283    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:11:52.005480 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.005390    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-cabundle-cert\") pod \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") "
Sep 09 20:11:52.005480 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.005431    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-proxy-tls\") pod \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") "
Sep 09 20:11:52.005480 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.005474    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8phzh\" (UniqueName: \"kubernetes.io/projected/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-api-access-8phzh\") pod \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") "
Sep 09 20:11:52.005773 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.005497    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-rbac-proxy-sar-config\") pod \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") "
Sep 09 20:11:52.005844 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.005703    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kserve-provision-location\") pod \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\" (UID: \"a3e6cb05-b02c-445a-8b0c-210f0dc29abf\") "
Sep 09 20:11:52.005924 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.005900    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3e6cb05-b02c-445a-8b0c-210f0dc29abf" (UID: "a3e6cb05-b02c-445a-8b0c-210f0dc29abf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:11:52.005924 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.005907    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3e6cb05-b02c-445a-8b0c-210f0dc29abf" (UID: "a3e6cb05-b02c-445a-8b0c-210f0dc29abf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:11:52.006125 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.006096    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-cabundle-cert\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.006125 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.006121    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.006469 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.006440    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3e6cb05-b02c-445a-8b0c-210f0dc29abf" (UID: "a3e6cb05-b02c-445a-8b0c-210f0dc29abf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:11:52.008047 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.008008    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3e6cb05-b02c-445a-8b0c-210f0dc29abf" (UID: "a3e6cb05-b02c-445a-8b0c-210f0dc29abf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:11:52.008146 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.008083    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-api-access-8phzh" (OuterVolumeSpecName: "kube-api-access-8phzh") pod "a3e6cb05-b02c-445a-8b0c-210f0dc29abf" (UID: "a3e6cb05-b02c-445a-8b0c-210f0dc29abf"). InnerVolumeSpecName "kube-api-access-8phzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:11:52.106957 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.106914    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-proxy-tls\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.106957 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.106947    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8phzh\" (UniqueName: \"kubernetes.io/projected/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kube-api-access-8phzh\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.106957 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.106960    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3e6cb05-b02c-445a-8b0c-210f0dc29abf-kserve-provision-location\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:11:52.351367 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.351280    2557 generic.go:356] "Generic (PLEG): container finished" podID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerID="d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a" exitCode=0
Sep 09 20:11:52.351367 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.351334    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"
Sep 09 20:11:52.351535 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.351375    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" event={"ID":"a3e6cb05-b02c-445a-8b0c-210f0dc29abf","Type":"ContainerDied","Data":"d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a"}
Sep 09 20:11:52.351535 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.351441    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22" event={"ID":"a3e6cb05-b02c-445a-8b0c-210f0dc29abf","Type":"ContainerDied","Data":"c387a8e3e6fbc9ec615c0ceb090a56b8a1650d2207e52ffe29cd4dbf9dfc0e1d"}
Sep 09 20:11:52.351535 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.351470    2557 scope.go:117] "RemoveContainer" containerID="4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a"
Sep 09 20:11:52.359855 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.359837    2557 scope.go:117] "RemoveContainer" containerID="d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a"
Sep 09 20:11:52.367908 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.367888    2557 scope.go:117] "RemoveContainer" containerID="ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c"
Sep 09 20:11:52.374465 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.374439    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"]
Sep 09 20:11:52.375585 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.375567    2557 scope.go:117] "RemoveContainer" containerID="4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a"
Sep 09 20:11:52.375907 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:11:52.375888    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a\": container with ID starting with 4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a not found: ID does not exist" containerID="4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a"
Sep 09 20:11:52.375951 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.375916    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a"} err="failed to get container status \"4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a\": rpc error: code = NotFound desc = could not find container \"4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a\": container with ID starting with 4761fa9938d5f4a9ed3b4c594ee6f5a5e6ab8552310a27c296a08c222b26098a not found: ID does not exist"
Sep 09 20:11:52.375951 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.375935    2557 scope.go:117] "RemoveContainer" containerID="d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a"
Sep 09 20:11:52.376205 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:11:52.376184    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a\": container with ID starting with d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a not found: ID does not exist" containerID="d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a"
Sep 09 20:11:52.376254 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.376210    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a"} err="failed to get container status \"d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a\": rpc error: code = NotFound desc = could not find container \"d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a\": container with ID starting with d89a8e94e03d3209260f1c94ebed97ef61c210c6d35249c379329a56642f551a not found: ID does not exist"
Sep 09 20:11:52.376254 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.376225    2557 scope.go:117] "RemoveContainer" containerID="ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c"
Sep 09 20:11:52.376483 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:11:52.376466    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c\": container with ID starting with ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c not found: ID does not exist" containerID="ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c"
Sep 09 20:11:52.376524 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.376492    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c"} err="failed to get container status \"ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c\": rpc error: code = NotFound desc = could not find container \"ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c\": container with ID starting with ecab9f2457fc9a587197163c72cc503fcc3ad6640cddaf8df53a356f6947259c not found: ID does not exist"
Sep 09 20:11:52.378605 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:52.378583    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-kdt22"]
Sep 09 20:11:53.637930 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:11:53.637896    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" path="/var/lib/kubelet/pods/a3e6cb05-b02c-445a-8b0c-210f0dc29abf/volumes"
Sep 09 20:12:00.169893 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:00.169852    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 20:12:10.170385 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:10.170349    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:12:27.624078 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:27.624041    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"]
Sep 09 20:12:27.624729 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:27.624310    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" containerID="cri-o://e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4" gracePeriod=30
Sep 09 20:12:27.624729 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:27.624377    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kube-rbac-proxy" containerID="cri-o://369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf" gracePeriod=30
Sep 09 20:12:28.489504 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:28.489468    2557 generic.go:356] "Generic (PLEG): container finished" podID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerID="369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf" exitCode=2
Sep 09 20:12:28.489672 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:28.489543    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" event={"ID":"5adce253-03bf-4f36-97c2-4e0b0be3b257","Type":"ContainerDied","Data":"369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf"}
Sep 09 20:12:30.164859 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:30.164817    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 09 20:12:30.169149 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:30.169121    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 09 20:12:31.370925 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.370899    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:12:31.502570 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.502536    2557 generic.go:356] "Generic (PLEG): container finished" podID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerID="e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4" exitCode=0
Sep 09 20:12:31.502740 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.502593    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"
Sep 09 20:12:31.502740 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.502620    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" event={"ID":"5adce253-03bf-4f36-97c2-4e0b0be3b257","Type":"ContainerDied","Data":"e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4"}
Sep 09 20:12:31.502740 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.502688    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp" event={"ID":"5adce253-03bf-4f36-97c2-4e0b0be3b257","Type":"ContainerDied","Data":"03e60ddcacadf0579d65ccf31368d4bd51de4e3f94d37f9fdda98f808e3166bb"}
Sep 09 20:12:31.502740 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.502716    2557 scope.go:117] "RemoveContainer" containerID="369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf"
Sep 09 20:12:31.510436 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.510420    2557 scope.go:117] "RemoveContainer" containerID="e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4"
Sep 09 20:12:31.517687 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.517661    2557 scope.go:117] "RemoveContainer" containerID="1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4"
Sep 09 20:12:31.524862 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.524845    2557 scope.go:117] "RemoveContainer" containerID="369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf"
Sep 09 20:12:31.525129 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:12:31.525111    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf\": container with ID starting with 369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf not found: ID does not exist" containerID="369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf"
Sep 09 20:12:31.525183 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.525139    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf"} err="failed to get container status \"369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf\": rpc error: code = NotFound desc = could not find container \"369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf\": container with ID starting with 369945820d16aa636926fc606685ab642739f5e7c5c3bb48038edd0fe6a02aaf not found: ID does not exist"
Sep 09 20:12:31.525183 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.525156    2557 scope.go:117] "RemoveContainer" containerID="e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4"
Sep 09 20:12:31.525417 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:12:31.525399    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4\": container with ID starting with e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4 not found: ID does not exist" containerID="e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4"
Sep 09 20:12:31.525457 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.525423    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4"} err="failed to get container status \"e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4\": rpc error: code = NotFound desc = could not find container \"e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4\": container with ID starting with e3f8b1ac4842ba32e74f5321d25ca12c8e74d8e317f5a9dc06dfb0606e7a6fe4 not found: ID does not exist"
Sep 09 20:12:31.525457 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.525436    2557 scope.go:117] "RemoveContainer" containerID="1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4"
Sep 09 20:12:31.525656 ip-10-0-143-121 kubenswrapper[2557]: E0909 20:12:31.525637    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4\": container with ID starting with 1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4 not found: ID does not exist" containerID="1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4"
Sep 09 20:12:31.525699 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.525663    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4"} err="failed to get container status \"1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4\": rpc error: code = NotFound desc = could not find container \"1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4\": container with ID starting with 1ae19bdc342309ccdf5aa7eeeb414d7f38836ed9989dd93a2d9ffbed4c21eee4 not found: ID does not exist"
Sep 09 20:12:31.533919 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.533899    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5adce253-03bf-4f36-97c2-4e0b0be3b257-kserve-provision-location\") pod \"5adce253-03bf-4f36-97c2-4e0b0be3b257\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") "
Sep 09 20:12:31.533976 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.533931    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-rbac-proxy-sar-config\") pod \"5adce253-03bf-4f36-97c2-4e0b0be3b257\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") "
Sep 09 20:12:31.534028 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.533980    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm5vh\" (UniqueName: \"kubernetes.io/projected/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-api-access-tm5vh\") pod \"5adce253-03bf-4f36-97c2-4e0b0be3b257\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") "
Sep 09 20:12:31.534081 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.534056    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5adce253-03bf-4f36-97c2-4e0b0be3b257-proxy-tls\") pod \"5adce253-03bf-4f36-97c2-4e0b0be3b257\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") "
Sep 09 20:12:31.534137 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.534087    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-cabundle-cert\") pod \"5adce253-03bf-4f36-97c2-4e0b0be3b257\" (UID: \"5adce253-03bf-4f36-97c2-4e0b0be3b257\") "
Sep 09 20:12:31.534309 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.534286    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5adce253-03bf-4f36-97c2-4e0b0be3b257-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5adce253-03bf-4f36-97c2-4e0b0be3b257" (UID: "5adce253-03bf-4f36-97c2-4e0b0be3b257"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:31.534412 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.534318    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5adce253-03bf-4f36-97c2-4e0b0be3b257" (UID: "5adce253-03bf-4f36-97c2-4e0b0be3b257"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:31.534518 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.534474    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5adce253-03bf-4f36-97c2-4e0b0be3b257" (UID: "5adce253-03bf-4f36-97c2-4e0b0be3b257"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:31.536028 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.536005    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5adce253-03bf-4f36-97c2-4e0b0be3b257-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5adce253-03bf-4f36-97c2-4e0b0be3b257" (UID: "5adce253-03bf-4f36-97c2-4e0b0be3b257"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:31.536171 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.536139    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-api-access-tm5vh" (OuterVolumeSpecName: "kube-api-access-tm5vh") pod "5adce253-03bf-4f36-97c2-4e0b0be3b257" (UID: "5adce253-03bf-4f36-97c2-4e0b0be3b257"). InnerVolumeSpecName "kube-api-access-tm5vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:31.634936 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.634900    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm5vh\" (UniqueName: \"kubernetes.io/projected/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-api-access-tm5vh\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:12:31.634936 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.634929    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5adce253-03bf-4f36-97c2-4e0b0be3b257-proxy-tls\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:12:31.634936 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.634944    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-cabundle-cert\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:12:31.635186 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.634955    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5adce253-03bf-4f36-97c2-4e0b0be3b257-kserve-provision-location\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:12:31.635186 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.634967    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5adce253-03bf-4f36-97c2-4e0b0be3b257-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-121.ec2.internal\" DevicePath \"\""
Sep 09 20:12:31.820838 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.820762    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"]
Sep 09 20:12:31.826643 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:31.826615    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-5164f-predictor-76497dcd57-hxhwp"]
Sep 09 20:12:33.642956 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:12:33.642920    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" path="/var/lib/kubelet/pods/5adce253-03bf-4f36-97c2-4e0b0be3b257/volumes"
Sep 09 20:24:35.639874 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.639841    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cwwj7/must-gather-cgxhp"]
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640219    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kube-rbac-proxy"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640238    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kube-rbac-proxy"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640257    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kube-rbac-proxy"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640265    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kube-rbac-proxy"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640275    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="storage-initializer"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640284    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="storage-initializer"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640294    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640303    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640319    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="storage-initializer"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640327    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="storage-initializer"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640336    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640344    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640439    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kserve-container"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640450    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kube-rbac-proxy"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640459    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e6cb05-b02c-445a-8b0c-210f0dc29abf" containerName="kube-rbac-proxy"
Sep 09 20:24:35.640486 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.640468    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5adce253-03bf-4f36-97c2-4e0b0be3b257" containerName="kserve-container"
Sep 09 20:24:35.643355 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.643336    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwwj7/must-gather-cgxhp"
Sep 09 20:24:35.645489 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.645466    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cwwj7\"/\"default-dockercfg-54whg\""
Sep 09 20:24:35.645665 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.645648    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cwwj7\"/\"openshift-service-ca.crt\""
Sep 09 20:24:35.645986 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.645973    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cwwj7\"/\"kube-root-ca.crt\""
Sep 09 20:24:35.651260 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.651240    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwwj7/must-gather-cgxhp"]
Sep 09 20:24:35.704754 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.704709    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c948dfe8-2f17-419d-8fa4-c7a30e995015-must-gather-output\") pod \"must-gather-cgxhp\" (UID: \"c948dfe8-2f17-419d-8fa4-c7a30e995015\") " pod="openshift-must-gather-cwwj7/must-gather-cgxhp"
Sep 09 20:24:35.704931 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.704769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5cl\" (UniqueName: \"kubernetes.io/projected/c948dfe8-2f17-419d-8fa4-c7a30e995015-kube-api-access-bz5cl\") pod \"must-gather-cgxhp\" (UID: \"c948dfe8-2f17-419d-8fa4-c7a30e995015\") " pod="openshift-must-gather-cwwj7/must-gather-cgxhp"
Sep 09 20:24:35.805370 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.805330    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c948dfe8-2f17-419d-8fa4-c7a30e995015-must-gather-output\") pod \"must-gather-cgxhp\" (UID: \"c948dfe8-2f17-419d-8fa4-c7a30e995015\") " pod="openshift-must-gather-cwwj7/must-gather-cgxhp"
Sep 09 20:24:35.805537 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.805386    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5cl\" (UniqueName: \"kubernetes.io/projected/c948dfe8-2f17-419d-8fa4-c7a30e995015-kube-api-access-bz5cl\") pod \"must-gather-cgxhp\" (UID: \"c948dfe8-2f17-419d-8fa4-c7a30e995015\") " pod="openshift-must-gather-cwwj7/must-gather-cgxhp"
Sep 09 20:24:35.805666 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.805647    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c948dfe8-2f17-419d-8fa4-c7a30e995015-must-gather-output\") pod \"must-gather-cgxhp\" (UID: \"c948dfe8-2f17-419d-8fa4-c7a30e995015\") " pod="openshift-must-gather-cwwj7/must-gather-cgxhp"
Sep 09 20:24:35.812981 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.812952    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5cl\" (UniqueName: \"kubernetes.io/projected/c948dfe8-2f17-419d-8fa4-c7a30e995015-kube-api-access-bz5cl\") pod \"must-gather-cgxhp\" (UID: \"c948dfe8-2f17-419d-8fa4-c7a30e995015\") " pod="openshift-must-gather-cwwj7/must-gather-cgxhp"
Sep 09 20:24:35.952428 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:35.952385    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwwj7/must-gather-cgxhp"
Sep 09 20:24:36.222260 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:36.222171    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwwj7/must-gather-cgxhp"]
Sep 09 20:24:36.225590 ip-10-0-143-121 kubenswrapper[2557]: W0909 20:24:36.225564    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc948dfe8_2f17_419d_8fa4_c7a30e995015.slice/crio-0fdcbcda3fb0c216ea91c1f920c5543165d422f2ed924050c3b0f90a5d5bd4c1 WatchSource:0}: Error finding container 0fdcbcda3fb0c216ea91c1f920c5543165d422f2ed924050c3b0f90a5d5bd4c1: Status 404 returned error can't find the container with id 0fdcbcda3fb0c216ea91c1f920c5543165d422f2ed924050c3b0f90a5d5bd4c1
Sep 09 20:24:36.227530 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:36.227511    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:24:37.157785 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:37.157747    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwwj7/must-gather-cgxhp" event={"ID":"c948dfe8-2f17-419d-8fa4-c7a30e995015","Type":"ContainerStarted","Data":"0fdcbcda3fb0c216ea91c1f920c5543165d422f2ed924050c3b0f90a5d5bd4c1"}
Sep 09 20:24:38.165912 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:38.165864    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwwj7/must-gather-cgxhp" event={"ID":"c948dfe8-2f17-419d-8fa4-c7a30e995015","Type":"ContainerStarted","Data":"ad3865e63c34ab19ab7c24f26ec204ff5f2012b2d25b8d92afe8cb5f158deff2"}
Sep 09 20:24:38.166397 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:38.165927    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwwj7/must-gather-cgxhp" event={"ID":"c948dfe8-2f17-419d-8fa4-c7a30e995015","Type":"ContainerStarted","Data":"680ab958cf3ed02184bbd4be584b47e2269beea6c59bc727c54a2f0cf0846212"}
Sep 09 20:24:38.182508 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:38.181036    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cwwj7/must-gather-cgxhp" podStartSLOduration=2.237756848 podStartE2EDuration="3.181013233s" totalImagesPullingTime="943.256385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:38.179898438 +0000 UTC m=+1751.075920303" watchObservedRunningTime="2026-09-09 20:24:38.181013233 +0000 UTC m=+1751.077035061"
Sep 09 20:24:38.604890 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:38.604812    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6jmkw_b1fb14a2-9c6e-455e-a59a-bde2e283c026/global-pull-secret-syncer/0.log"
Sep 09 20:24:38.792604 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:38.792566    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xzgsg_d29c4774-ef1c-46f7-b0c6-82ea8fdebaa2/konnectivity-agent/0.log"
Sep 09 20:24:38.867292 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:38.867218    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-121.ec2.internal_efd24342e66e0564f736a8b8390c0964/haproxy/0.log"
Sep 09 20:24:43.413406 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:43.413370    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-dxql5_9f21e8ad-2bb4-49ec-bc5d-72e1abd32773/cluster-monitoring-operator/0.log"
Sep 09 20:24:43.553315 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:43.553287    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-cwn5h_df4193bb-cba4-44d3-8b1f-247563b1b036/monitoring-plugin/0.log"
Sep 09 20:24:43.674074 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:43.674039    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cv9h_2d1a993c-544d-4bf8-a879-0ddf547030d6/node-exporter/0.log"
Sep 09 20:24:43.702190 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:43.702141    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cv9h_2d1a993c-544d-4bf8-a879-0ddf547030d6/kube-rbac-proxy/0.log"
Sep 09 20:24:43.732185 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:43.732024    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cv9h_2d1a993c-544d-4bf8-a879-0ddf547030d6/init-textfile/0.log"
Sep 09 20:24:43.960061 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:43.959973    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81d4b79e-d4f7-4f2c-96f6-21a5070347a0/prometheus/0.log"
Sep 09 20:24:43.979978 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:43.979942    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81d4b79e-d4f7-4f2c-96f6-21a5070347a0/config-reloader/0.log"
Sep 09 20:24:44.004058 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:44.004026    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81d4b79e-d4f7-4f2c-96f6-21a5070347a0/thanos-sidecar/0.log"
Sep 09 20:24:44.028557 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:44.028528    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81d4b79e-d4f7-4f2c-96f6-21a5070347a0/kube-rbac-proxy-web/0.log"
Sep 09 20:24:44.059121 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:44.059083    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81d4b79e-d4f7-4f2c-96f6-21a5070347a0/kube-rbac-proxy/0.log"
Sep 09 20:24:44.082218 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:44.082188    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81d4b79e-d4f7-4f2c-96f6-21a5070347a0/kube-rbac-proxy-thanos/0.log"
Sep 09 20:24:44.111676 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:44.111646    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81d4b79e-d4f7-4f2c-96f6-21a5070347a0/init-config-reloader/0.log"
Sep 09 20:24:46.810304 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:46.810267    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"]
Sep 09 20:24:46.815482 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:46.815452    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:46.823625 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:46.823593    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"]
Sep 09 20:24:46.907372 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:46.907327    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-lib-modules\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:46.907630 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:46.907608    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-proc\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:46.907799 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:46.907776    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-podres\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:46.907913 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:46.907832    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thtw\" (UniqueName: \"kubernetes.io/projected/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-kube-api-access-9thtw\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:46.907978 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:46.907941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-sys\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.009058 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.009021    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-lib-modules\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.009058 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.009059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-proc\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.009306 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.009084    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-podres\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.009306 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.009124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thtw\" (UniqueName: \"kubernetes.io/projected/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-kube-api-access-9thtw\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.009306 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.009182    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-sys\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.009306 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.009204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-proc\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.009306 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.009232    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-lib-modules\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.009306 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.009263    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-sys\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.009517 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.009309    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-podres\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.018410 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.018369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thtw\" (UniqueName: \"kubernetes.io/projected/9c5a8ef8-6fa1-4495-9407-de43c4d78a1f-kube-api-access-9thtw\") pod \"perf-node-gather-daemonset-gcvwn\" (UID: \"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f\") " pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.128641 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.128553    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:47.295607 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.295575    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-xxzd8_35446b71-e1a6-47ab-b38e-1daecf7b0ad2/volume-data-source-validator/0.log"
Sep 09 20:24:47.416869 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:47.416834    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"]
Sep 09 20:24:47.421031 ip-10-0-143-121 kubenswrapper[2557]: W0909 20:24:47.420958    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9c5a8ef8_6fa1_4495_9407_de43c4d78a1f.slice/crio-fda0f9e2802a0af167f29468b0f043e88fcbacd9b25ccdedebdc56d5e0440ce8 WatchSource:0}: Error finding container fda0f9e2802a0af167f29468b0f043e88fcbacd9b25ccdedebdc56d5e0440ce8: Status 404 returned error can't find the container with id fda0f9e2802a0af167f29468b0f043e88fcbacd9b25ccdedebdc56d5e0440ce8
Sep 09 20:24:48.265508 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:48.265471    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn" event={"ID":"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f","Type":"ContainerStarted","Data":"60dd88fcdb4fad8263b87f2b43ac2bd1f0f30cf644f94e37f0a76b0703436127"}
Sep 09 20:24:48.265927 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:48.265532    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn" event={"ID":"9c5a8ef8-6fa1-4495-9407-de43c4d78a1f","Type":"ContainerStarted","Data":"fda0f9e2802a0af167f29468b0f043e88fcbacd9b25ccdedebdc56d5e0440ce8"}
Sep 09 20:24:48.265927 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:48.265559    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:48.282789 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:48.282725    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn" podStartSLOduration=2.282702219 podStartE2EDuration="2.282702219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:48.280811157 +0000 UTC m=+1761.176832994" watchObservedRunningTime="2026-09-09 20:24:48.282702219 +0000 UTC m=+1761.178724047"
Sep 09 20:24:48.345213 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:48.345117    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sm8l8_2c0e6cc9-602e-4752-92b2-e32a62763c2a/dns/0.log"
Sep 09 20:24:48.367467 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:48.367421    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sm8l8_2c0e6cc9-602e-4752-92b2-e32a62763c2a/kube-rbac-proxy/0.log"
Sep 09 20:24:48.453054 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:48.453016    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9vl9_7d393eee-6f0d-4ca1-bae0-2d669406772c/dns-node-resolver/0.log"
Sep 09 20:24:48.994261 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:48.994231    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5d8wp_ec73c477-ff0a-4794-94d2-a905c2c7ba37/node-ca/0.log"
Sep 09 20:24:50.168715 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:50.168660    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-425sq_ab31a95f-faaf-45ea-9290-95198e492b95/serve-healthcheck-canary/0.log"
Sep 09 20:24:50.607898 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:50.607818    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-mglmh_7fa75e95-b8a6-4afe-a78d-f405f9cc3212/insights-operator/0.log"
Sep 09 20:24:50.608120 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:50.608102    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-mglmh_7fa75e95-b8a6-4afe-a78d-f405f9cc3212/insights-operator/1.log"
Sep 09 20:24:50.706112 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:50.706085    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ms748_fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9/kube-rbac-proxy/0.log"
Sep 09 20:24:50.728724 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:50.728695    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ms748_fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9/exporter/0.log"
Sep 09 20:24:50.750052 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:50.750028    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ms748_fc0fdb1e-8e87-4c6c-bce4-b4be82ed8ec9/extractor/0.log"
Sep 09 20:24:53.338209 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:53.338156    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x8t7v_9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f/s3-init/0.log"
Sep 09 20:24:53.356661 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:53.356626    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x8t7v_9bb10c96-c99e-4efb-8a1d-4fad7c9fb78f/download-hf-model/0.log"
Sep 09 20:24:54.279290 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:54.279256    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cwwj7/perf-node-gather-daemonset-gcvwn"
Sep 09 20:24:58.151733 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:58.151702    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-tcrdt_2aea9318-4fda-49f0-bc0f-39d3c5165401/kube-storage-version-migrator-operator/1.log"
Sep 09 20:24:58.152569 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:58.152544    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-tcrdt_2aea9318-4fda-49f0-bc0f-39d3c5165401/kube-storage-version-migrator-operator/0.log"
Sep 09 20:24:59.161916 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.161887    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59wff_a0d2e226-3707-4893-b20e-1686fddccf6b/kube-multus-additional-cni-plugins/0.log"
Sep 09 20:24:59.184387 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.184339    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59wff_a0d2e226-3707-4893-b20e-1686fddccf6b/egress-router-binary-copy/0.log"
Sep 09 20:24:59.208750 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.208720    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59wff_a0d2e226-3707-4893-b20e-1686fddccf6b/cni-plugins/0.log"
Sep 09 20:24:59.230419 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.230392    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59wff_a0d2e226-3707-4893-b20e-1686fddccf6b/bond-cni-plugin/0.log"
Sep 09 20:24:59.254131 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.254104    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59wff_a0d2e226-3707-4893-b20e-1686fddccf6b/routeoverride-cni/0.log"
Sep 09 20:24:59.275628 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.275601    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59wff_a0d2e226-3707-4893-b20e-1686fddccf6b/whereabouts-cni-bincopy/0.log"
Sep 09 20:24:59.298913 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.298887    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-59wff_a0d2e226-3707-4893-b20e-1686fddccf6b/whereabouts-cni/0.log"
Sep 09 20:24:59.694712 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.694675    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-f4pbr_6ea52b7e-e5d7-4053-b28d-7a9c6410dc1f/kube-multus/0.log"
Sep 09 20:24:59.819856 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.819826    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2mfbb_31bdfd97-00d0-4a49-8b1f-8bd09ad7df87/network-metrics-daemon/0.log"
Sep 09 20:24:59.841212 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:24:59.841177    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2mfbb_31bdfd97-00d0-4a49-8b1f-8bd09ad7df87/kube-rbac-proxy/0.log"
Sep 09 20:25:01.145881 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:01.145847    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smhwd_4176db56-adb0-4a49-bcd3-6720c27d4dbe/ovn-controller/0.log"
Sep 09 20:25:01.162525 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:01.162500    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smhwd_4176db56-adb0-4a49-bcd3-6720c27d4dbe/ovn-acl-logging/0.log"
Sep 09 20:25:01.185871 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:01.185846    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smhwd_4176db56-adb0-4a49-bcd3-6720c27d4dbe/kube-rbac-proxy-node/0.log"
Sep 09 20:25:01.208434 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:01.208397    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smhwd_4176db56-adb0-4a49-bcd3-6720c27d4dbe/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 20:25:01.227131 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:01.227109    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smhwd_4176db56-adb0-4a49-bcd3-6720c27d4dbe/northd/0.log"
Sep 09 20:25:01.255367 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:01.255335    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smhwd_4176db56-adb0-4a49-bcd3-6720c27d4dbe/nbdb/0.log"
Sep 09 20:25:01.276431 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:01.276394    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smhwd_4176db56-adb0-4a49-bcd3-6720c27d4dbe/sbdb/0.log"
Sep 09 20:25:01.313750 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:01.313721    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-smhwd_4176db56-adb0-4a49-bcd3-6720c27d4dbe/ovnkube-controller/0.log"
Sep 09 20:25:02.357715 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:02.357684    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dfq9t_1c4069a3-7ecc-4569-b976-3c59215327b2/network-check-target-container/0.log"
Sep 09 20:25:03.448747 ip-10-0-143-121 kubenswrapper[2557]: I0909 20:25:03.448715    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kn5c2_e0bae31c-37ce-4f6c-8ece-a8f3be26cd45/iptables-alerter/0.log"