Sep 09 14:33:20.212621 ip-10-0-130-173 systemd[1]: Starting Kubernetes Kubelet... Sep 09 14:33:20.634279 ip-10-0-130-173 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 14:33:20.634279 ip-10-0-130-173 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 14:33:20.634279 ip-10-0-130-173 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 14:33:20.634279 ip-10-0-130-173 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 14:33:20.634279 ip-10-0-130-173 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 14:33:20.635729 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.635643 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 14:33:20.640780 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640764 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:20.640780 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640780 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640784 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640787 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640802 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640805 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640808 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640811 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640814 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640816 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640819 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640822 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640825 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640828 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640830 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640833 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640835 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640838 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640841 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640843 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640846 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:20.640849 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640848 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640851 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640853 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640856 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640859 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640862 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640865 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640867 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640870 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640872 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640875 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640877 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640881 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640884 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640887 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640890 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640893 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640897 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640899 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640902 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:20.641325 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640904 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640907 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640909 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640911 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640914 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640917 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640919 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640922 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640924 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640926 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640929 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640931 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640934 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640936 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640939 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640942 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640945 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640947 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640950 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640952 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:20.641865 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640955 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640959 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640961 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640964 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640967 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640969 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640971 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640974 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640976 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640979 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640982 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640985 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640987 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640990 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640993 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.640997 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641000 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641003 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641006 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641009 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:20.642448 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641011 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641014 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641016 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641019 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641021 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641025 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641029 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641032 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641036 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641039 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641042 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641044 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641047 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641050 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641052 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641055 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641057 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641060 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641780 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:20.642925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641789 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641792 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641795 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641800 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641805 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641808 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641811 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641814 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641817 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641820 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641822 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641825 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641828 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641830 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641833 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641836 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641840 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641843 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641846 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:20.643409 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641849 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641852 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641855 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641858 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641861 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641864 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641866 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641869 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641871 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641874 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641877 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641880 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641882 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641885 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641887 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641890 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641892 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641895 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641898 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641901 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:20.643883 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641903 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641905 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641908 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641911 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641913 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641916 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641918 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641921 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641923 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641926 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641928 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641932 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641935 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641937 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641940 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641943 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641945 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641948 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641950 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641953 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:20.644422 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641955 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641958 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641960 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641963 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641965 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641968 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641970 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641972 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641975 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641977 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641980 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641983 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641986 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641989 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641992 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641994 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641997 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.641999 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642003 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642007 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:20.644919 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642010 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642012 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642015 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642018 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642023 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642026 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642028 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642031 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642033 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642036 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642039 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642041 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642044 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642046 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642048 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642051 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642053 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642056 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.642059 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643279 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:20.645428 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643286 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643290 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643294 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643297 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643300 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643302 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643305 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643310 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643313 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643316 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643320 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643323 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643327 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643329 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643332 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643336 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643338 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643342 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643357 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:20.645916 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643360 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643362 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643365 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643367 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643370 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643373 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643375 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643378 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643380 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643383 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643386 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643388 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643391 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643394 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643396 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643399 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643404 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643407 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643409 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643412 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:20.646406 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643415 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643417 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643420 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643423 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643426 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643429 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643433 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643436 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643440 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643442 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643445 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643448 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643450 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643454 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643457 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643460 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643463 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643465 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643469 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:20.646887 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643471 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643474 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643476 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643479 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643482 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643484 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643487 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643490 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643492 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643494 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643497 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643499 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643502 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643505 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643507 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643509 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643512 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643515 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643517 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643520 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:20.647375 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643523 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643526 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643529 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643531 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643533 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643536 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643538 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643541 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643543 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643546 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643548 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643551 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643553 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643556 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643558 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643560 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643563 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643565 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643568 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.643571 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643633 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 14:33:20.647871 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643662 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643668 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643673 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643677 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643680 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643684 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643688 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643692 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643695 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643699 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643702 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643705 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643708 2578 flags.go:64] FLAG: --cgroup-root="" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643711 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643714 2578 flags.go:64] FLAG: --client-ca-file="" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643717 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643719 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643724 2578 flags.go:64] FLAG: --cluster-domain="" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643727 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643730 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643733 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643736 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643740 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 14:33:20.648386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643743 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643746 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643749 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643752 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643754 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643758 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643761 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643763 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643767 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643770 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643773 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643775 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643778 2578 flags.go:64] FLAG: --enable-server="true" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643781 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643785 2578 flags.go:64] FLAG: --event-burst="100" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643788 2578 flags.go:64] FLAG: --event-qps="50" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643791 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643797 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643799 2578 flags.go:64] FLAG: --eviction-hard="" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643803 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643805 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643809 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643812 2578 flags.go:64] FLAG: --eviction-soft="" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643815 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643818 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 14:33:20.648939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643821 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643824 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643826 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643829 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643832 2578 flags.go:64] FLAG: --feature-gates="" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643836 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643839 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643842 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643845 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643848 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643851 2578 flags.go:64] FLAG: --help="false" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643854 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-130-173.ec2.internal" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643857 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643859 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643862 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643865 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643868 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643871 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643874 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643877 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643880 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643883 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643886 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643890 2578 flags.go:64] FLAG: --kube-reserved="" Sep 09 14:33:20.649603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643894 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643896 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643899 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643902 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643907 2578 flags.go:64] FLAG: --lock-file="" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643910 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643912 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643915 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643920 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643923 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643926 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643939 2578 flags.go:64] FLAG: --logging-format="text" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643942 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643974 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643979 2578 flags.go:64] FLAG: --manifest-url="" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643984 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643990 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643994 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.643998 2578 flags.go:64] FLAG: --max-pods="110" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644002 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644005 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644008 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644011 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644015 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644018 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 14:33:20.650176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644021 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644028 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644032 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644035 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644038 2578 flags.go:64] FLAG: --pod-cidr="" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644041 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644047 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644050 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644053 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644056 2578 flags.go:64] FLAG: --port="10250" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644060 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644063 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a5ec4900d4aa205d" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644066 2578 flags.go:64] FLAG: --qos-reserved="" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644071 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644074 2578 flags.go:64] FLAG: --register-node="true" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644077 2578 flags.go:64] FLAG: --register-with-taints="" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644081 2578 flags.go:64] FLAG: --registry-burst="10" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644083 2578 flags.go:64] FLAG: --registry-qps="5" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644086 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644089 2578 flags.go:64] FLAG: --reserved-memory="" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644094 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644097 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644100 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644103 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644106 2578 flags.go:64] FLAG: --runonce="false" Sep 09 14:33:20.650825 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644109 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644112 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644115 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644119 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644122 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644125 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644128 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644131 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644134 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644137 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644140 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644143 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644146 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644149 2578 flags.go:64] FLAG: --system-cgroups="" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644151 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644157 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644160 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644163 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644169 2578 flags.go:64] FLAG: --tls-min-version="" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644171 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644174 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644178 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644181 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644184 2578 flags.go:64] FLAG: --v="2" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644188 2578 flags.go:64] FLAG: --version="false" Sep 09 14:33:20.651463 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644192 2578 flags.go:64] FLAG: --vmodule="" Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644196 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.644199 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644321 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644325 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644329 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644331 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644334 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644338 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644341 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644354 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644357 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644360 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644363 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644365 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644368 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644371 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644374 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644377 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644379 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:20.652063 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644382 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644384 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644387 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644389 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644392 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644394 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644397 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644399 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644402 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644405 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644408 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644410 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644413 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644416 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644418 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644424 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644427 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644429 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644432 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644435 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:20.652589 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644438 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644441 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644443 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644447 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644449 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644452 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644455 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644457 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644460 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644463 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644465 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644468 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644471 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644473 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644476 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644478 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644481 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644484 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644486 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644489 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:20.653431 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644492 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644494 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644497 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644500 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644502 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644521 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644525 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644529 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644532 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644535 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644538 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644542 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644545 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644547 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644550 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644552 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644555 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644557 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644560 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644563 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:20.654148 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644566 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644568 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644571 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644574 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644576 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644579 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644583 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644586 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644589 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644592 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644595 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644598 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644601 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644604 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644608 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644611 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644614 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644617 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644620 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:20.654658 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644623 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:20.655118 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644626 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:20.655118 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.644630 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:20.655118 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.646030 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 14:33:20.655118 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.654679 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 14:33:20.655118 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.654695 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 14:33:20.655499 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.655489 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 14:33:20.655529 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.655503 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 14:33:20.655564 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655560 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655566 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655570 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655573 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655577 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655580 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655583 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655586 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655588 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655591 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655594 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655597 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:20.655594 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655600 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655603 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655606 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655608 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655611 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655614 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655617 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655620 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655622 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655625 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655628 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655630 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655633 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655635 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655638 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655640 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655643 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655645 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655648 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655651 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:20.655885 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655654 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655658 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655661 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655663 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655666 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655668 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655671 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655673 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655676 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655678 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655680 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655683 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655687 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655690 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655694 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655697 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655700 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655703 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655706 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:20.656460 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655709 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655712 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655715 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655718 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655721 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655724 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655726 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655729 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655731 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655734 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655736 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655739 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655742 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655745 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655748 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655750 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655753 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655756 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655758 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655760 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:20.656925 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655763 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655766 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655770 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655773 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655776 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655780 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655782 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655785 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655789 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655792 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655795 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655797 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655800 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655802 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655805 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655808 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655810 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655813 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655815 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:20.657429 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655818 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655821 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655823 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655826 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655829 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655831 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655834 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655837 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655839 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.655844 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655963 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655968 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655971 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655974 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655977 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 14:33:20.657929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655979 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655982 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655985 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655987 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655991 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655993 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655996 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.655999 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656002 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656005 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656008 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656010 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656013 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656016 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656018 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656021 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656023 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656027 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656030 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 14:33:20.658308 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656033 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656036 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656038 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656041 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656045 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656048 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656052 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656054 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656057 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656060 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656063 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656066 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656068 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656071 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656073 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656076 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656078 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656082 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656085 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 14:33:20.658781 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656087 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656090 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656093 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656096 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656099 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656101 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656104 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656106 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656108 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656111 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656113 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656116 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656118 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656121 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656123 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656126 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656128 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656131 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656133 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 14:33:20.659226 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656135 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656138 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656141 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656143 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656145 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656148 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656151 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656153 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656156 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656158 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656161 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656164 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656166 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656169 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656171 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656174 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656177 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656180 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656182 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656185 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 14:33:20.659694 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656187 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656190 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656192 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656195 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656197 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656199 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656202 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656205 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656207 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656209 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656213 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656216 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656219 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656222 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656225 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656227 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 14:33:20.660179 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:20.656230 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 14:33:20.660578 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.656235 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 14:33:20.660578 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.656340 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 14:33:20.660578 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.658436 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 14:33:20.660578 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.659420 2578 server.go:1019] "Starting client certificate rotation" Sep 09 14:33:20.660578 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.659526 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 14:33:20.660578 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.659562 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 14:33:20.662019 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.662005 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 14:33:20.663096 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.663080 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 14:33:20.674394 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.674371 2578 log.go:25] "Validated CRI v1 runtime API" Sep 09 14:33:20.677483 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.677469 2578 log.go:25] "Validated CRI v1 image API" Sep 09 14:33:20.680231 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.680215 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 14:33:20.683312 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.683292 2578 fs.go:135] Filesystem UUIDs: map[67e27e03-2d95-4e9d-beb3-c2d9292d0cf2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 92daa6fd-70c9-490f-b0eb-95f371f7f709:/dev/nvme0n1p4] Sep 09 14:33:20.683402 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.683310 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 14:33:20.688386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.688262 2578 manager.go:216] Machine: {Timestamp:2026-09-09 14:33:20.686892816 +0000 UTC m=+0.353698853 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097795 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2599fdf01b095ffd0d07fa0d0f93ac SystemUUID:ec2599fd-f01b-095f-fd0d-07fa0d0f93ac BootID:4baded2d-c2bf-4c50-9428-d28871065d54 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d8:b3:8a:c1:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d8:b3:8a:c1:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:6a:c8:1f:31:7c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 14:33:20.688386 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.688380 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 14:33:20.688488 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.688458 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 14:33:20.689738 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.689722 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 14:33:20.691858 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.691835 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 14:33:20.692728 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.691862 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 14:33:20.693692 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.693666 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 14:33:20.693692 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.693691 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 14:33:20.693816 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.693702 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 14:33:20.693816 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.693722 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 14:33:20.695414 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.695395 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 14:33:20.699306 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.699294 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 09 14:33:20.699435 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.699425 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 14:33:20.699480 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.699454 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 09 14:33:20.699480 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.699465 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 14:33:20.699547 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.699483 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 14:33:20.699547 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.699495 2578 kubelet.go:398] "Adding apiserver pod source" Sep 09 14:33:20.699547 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.699504 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 14:33:20.699885 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.699873 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 14:33:20.699917 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.699898 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 14:33:20.702326 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.702301 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 14:33:20.704470 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.704455 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 14:33:20.704547 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.704493 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 14:33:20.704737 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.704724 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 14:33:20.704799 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.704743 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 14:33:20.704799 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.704753 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 14:33:20.704799 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.704761 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 14:33:20.704799 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.704772 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 14:33:20.704799 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.704781 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 14:33:20.705390 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.705378 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 14:33:20.705452 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.705393 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 14:33:20.705452 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.705404 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 14:33:20.705452 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.705413 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 14:33:20.705452 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.705425 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 14:33:20.705452 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.705438 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 14:33:20.706214 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.706203 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 14:33:20.706267 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.706216 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 14:33:20.709068 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.709048 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-173.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 14:33:20.709144 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.709071 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 14:33:20.709144 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.709068 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 14:33:20.709974 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.709959 2578 server.go:1268] "Started kubelet" Sep 09 14:33:20.710090 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.710060 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 14:33:20.710182 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.710062 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 14:33:20.710182 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.710146 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 14:33:20.710923 ip-10-0-130-173 systemd[1]: Started Kubernetes Kubelet. Sep 09 14:33:20.711405 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.711388 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 14:33:20.712379 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.712364 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 09 14:33:20.714881 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.714854 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 14:33:20.716544 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.716532 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 14:33:20.717215 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.717195 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 14:33:20.717331 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.717311 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-173.ec2.internal\" not found" Sep 09 14:33:20.717926 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.717904 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 14:33:20.718006 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.717941 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 14:33:20.718006 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.717907 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 14:33:20.718098 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.718009 2578 factory.go:55] Registering systemd factory Sep 09 14:33:20.718098 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.718024 2578 factory.go:223] Registration of the systemd container factory successfully Sep 09 14:33:20.718098 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.718073 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 09 14:33:20.718098 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.718082 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 09 14:33:20.718673 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.718661 2578 factory.go:153] Registering CRI-O factory Sep 09 14:33:20.718673 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.718673 2578 factory.go:223] Registration of the crio container factory successfully Sep 09 14:33:20.718740 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.718711 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 14:33:20.718740 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.718725 2578 factory.go:103] Registering Raw factory Sep 09 14:33:20.718740 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.718737 2578 manager.go:1196] Started watching for new ooms in manager Sep 09 14:33:20.719094 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.719081 2578 manager.go:349] Starting recovery of all containers Sep 09 14:33:20.719532 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.717867 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e4229cf2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.709934322 +0000 UTC m=+0.376740360,LastTimestamp:2026-09-09 14:33:20.709934322 +0000 UTC m=+0.376740360,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.722483 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.722294 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 14:33:20.722598 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.722308 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 14:33:20.730025 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.730008 2578 manager.go:354] Recovery completed Sep 09 14:33:20.734418 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.734405 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:20.736418 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.736402 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:20.736475 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.736432 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:20.736475 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.736442 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:20.736983 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.736971 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 14:33:20.736983 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.736982 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 14:33:20.737116 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.736983 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:20.737116 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.736996 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 09 14:33:20.737735 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.737719 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:20.737838 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.737744 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:20.737838 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.737754 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:20.738624 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.738547 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e5b6ba46 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-173.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.736418374 +0000 UTC m=+0.403224414,LastTimestamp:2026-09-09 14:33:20.736418374 +0000 UTC m=+0.403224414,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.739024 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.739012 2578 policy_none.go:49] "None policy: Start" Sep 09 14:33:20.739082 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.739029 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 14:33:20.739082 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.739039 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 14:33:20.740804 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.740792 2578 policy_none.go:47] "Start" Sep 09 14:33:20.746253 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.746191 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e5b703b5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-173.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.736437173 +0000 UTC m=+0.403243210,LastTimestamp:2026-09-09 14:33:20.736437173 +0000 UTC m=+0.403243210,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.749242 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.749184 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e5b72930 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-173.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.736446768 +0000 UTC m=+0.403252805,LastTimestamp:2026-09-09 14:33:20.736446768 +0000 UTC m=+0.403252805,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.756758 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.756685 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-173.ec2.internal.18d3ada4e5b6ba46\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e5b6ba46 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-173.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.736418374 +0000 UTC m=+0.403224414,LastTimestamp:2026-09-09 14:33:20.737732028 +0000 UTC m=+0.404538065,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.760407 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.760297 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-173.ec2.internal.18d3ada4e5b703b5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e5b703b5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-173.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.736437173 +0000 UTC m=+0.403243210,LastTimestamp:2026-09-09 14:33:20.737749097 +0000 UTC m=+0.404555134,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.766249 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.766193 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-173.ec2.internal.18d3ada4e5b72930\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e5b72930 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-173.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.736446768 +0000 UTC m=+0.403252805,LastTimestamp:2026-09-09 14:33:20.737758389 +0000 UTC m=+0.404564426,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.777565 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.777552 2578 manager.go:337] "Starting Device Plugin manager" Sep 09 14:33:20.777628 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.777595 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 14:33:20.777628 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.777604 2578 server.go:87] "Starting device plugin registration server" Sep 09 14:33:20.777891 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.777877 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 14:33:20.777977 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.777892 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 14:33:20.778033 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.777979 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 14:33:20.778078 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.778052 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 14:33:20.778078 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.778062 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 14:33:20.778678 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.778662 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 14:33:20.778750 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.778712 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-173.ec2.internal\" not found" Sep 09 14:33:20.797765 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.797708 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e94074c2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.795776194 +0000 UTC m=+0.462582234,LastTimestamp:2026-09-09 14:33:20.795776194 +0000 UTC m=+0.462582234,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.807243 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.807226 2578 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2611 into the system container \"/system.slice\": " Sep 09 14:33:20.813097 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.813080 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7cll5" Sep 09 14:33:20.820459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.820441 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7cll5" Sep 09 14:33:20.878724 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.878706 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:20.879834 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.879817 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:20.879903 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.879846 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:20.879903 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.879857 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:20.879903 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.879886 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-173.ec2.internal" Sep 09 14:33:20.888981 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.888865 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-173.ec2.internal.18d3ada4e5b6ba46\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e5b6ba46 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-173.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.736418374 +0000 UTC m=+0.403224414,LastTimestamp:2026-09-09 14:33:20.879833274 +0000 UTC m=+0.546639311,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.891656 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.891587 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-173.ec2.internal.18d3ada4e5b703b5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-173.ec2.internal.18d3ada4e5b703b5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-173.ec2.internal,UID:ip-10-0-130-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-173.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-173.ec2.internal,},FirstTimestamp:2026-09-09 14:33:20.736437173 +0000 UTC m=+0.403243210,LastTimestamp:2026-09-09 14:33:20.879850792 +0000 UTC m=+0.546656829,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-173.ec2.internal,}" Sep 09 14:33:20.891776 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.891732 2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-130-173.ec2.internal" Sep 09 14:33:20.927820 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.927798 2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-173.ec2.internal\" not found" node="ip-10-0-130-173.ec2.internal" Sep 09 14:33:20.974716 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.974676 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 14:33:20.975799 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.975781 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 14:33:20.975860 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.975836 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 14:33:20.975912 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.975862 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 14:33:20.975912 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:20.975905 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 14:33:20.978003 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:20.977986 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:21.076134 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.076099 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal"] Sep 09 14:33:21.076241 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.076169 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:21.077054 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.077039 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:21.077137 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.077066 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:21.077137 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.077077 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:21.079210 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.079198 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:21.079327 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.079313 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.079380 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.079359 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:21.079897 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.079881 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:21.079981 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.079897 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:21.079981 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.079909 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:21.079981 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.079918 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:21.079981 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.079922 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:21.079981 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.079936 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:21.082143 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.082127 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.082222 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.082157 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:21.082800 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.082783 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:21.082866 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.082817 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:21.082866 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.082832 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:21.092278 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.092263 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 14:33:21.092880 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.092867 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientMemory" Sep 09 14:33:21.092942 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.092895 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 14:33:21.092942 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.092910 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeHasSufficientPID" Sep 09 14:33:21.092942 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.092929 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.098938 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.098926 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.098980 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:21.098944 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-173.ec2.internal\": node \"ip-10-0-130-173.ec2.internal\" not found" Sep 09 14:33:21.112937 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:21.112914 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-173.ec2.internal\" not found" node="ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.117074 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:21.117054 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-173.ec2.internal\" not found" Sep 09 14:33:21.117239 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:21.117228 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-173.ec2.internal\" not found" node="ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.119281 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.119265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfc2c7a82d8ba99d752d94210c868394-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal\" (UID: \"dfc2c7a82d8ba99d752d94210c868394\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.119326 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.119289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfc2c7a82d8ba99d752d94210c868394-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal\" (UID: \"dfc2c7a82d8ba99d752d94210c868394\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.119326 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.119311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce69d8fb05d3b4f877b142f86f624c3c-config\") pod \"kube-apiserver-proxy-ip-10-0-130-173.ec2.internal\" (UID: \"ce69d8fb05d3b4f877b142f86f624c3c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.217747 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:21.217670 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-173.ec2.internal\" not found" Sep 09 14:33:21.220107 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.220091 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfc2c7a82d8ba99d752d94210c868394-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal\" (UID: \"dfc2c7a82d8ba99d752d94210c868394\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.220159 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.220118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfc2c7a82d8ba99d752d94210c868394-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal\" (UID: \"dfc2c7a82d8ba99d752d94210c868394\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.220159 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.220135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce69d8fb05d3b4f877b142f86f624c3c-config\") pod \"kube-apiserver-proxy-ip-10-0-130-173.ec2.internal\" (UID: \"ce69d8fb05d3b4f877b142f86f624c3c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.220225 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.220198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dfc2c7a82d8ba99d752d94210c868394-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal\" (UID: \"dfc2c7a82d8ba99d752d94210c868394\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.220225 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.220211 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce69d8fb05d3b4f877b142f86f624c3c-config\") pod \"kube-apiserver-proxy-ip-10-0-130-173.ec2.internal\" (UID: \"ce69d8fb05d3b4f877b142f86f624c3c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.220285 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.220212 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfc2c7a82d8ba99d752d94210c868394-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal\" (UID: \"dfc2c7a82d8ba99d752d94210c868394\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.318175 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:21.318145 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-173.ec2.internal\" not found" Sep 09 14:33:21.421747 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:21.421708 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-173.ec2.internal\" not found" Sep 09 14:33:21.421747 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.421727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.421912 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.421792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.522950 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:21.522863 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-173.ec2.internal\" not found" Sep 09 14:33:21.624891 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:21.624855 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-173.ec2.internal\" not found" Sep 09 14:33:21.631160 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.631140 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:21.659051 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.659020 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 14:33:21.659459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.659134 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 14:33:21.659459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.659183 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 14:33:21.718318 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.718291 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:21.818815 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.818732 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.830635 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.830617 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 14:33:21.831683 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.831670 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal" Sep 09 14:33:21.840787 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.840769 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 14:33:21.895593 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.895562 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 14:28:21 +0000 UTC" deadline="2028-02-05 09:48:31.232321044 +0000 UTC" Sep 09 14:33:21.895593 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:21.895590 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12331h15m9.336734725s" Sep 09 14:33:22.165449 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.165372 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:22.214732 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:22.214701 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc2c7a82d8ba99d752d94210c868394.slice/crio-455ff97ecd6556557e57106c959f19b23fbbab5cbd56de4c3d256621d6fc3550 WatchSource:0}: Error finding container 455ff97ecd6556557e57106c959f19b23fbbab5cbd56de4c3d256621d6fc3550: Status 404 returned error can't find the container with id 455ff97ecd6556557e57106c959f19b23fbbab5cbd56de4c3d256621d6fc3550 Sep 09 14:33:22.219112 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.219091 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 14:33:22.515744 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.515665 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:22.665649 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:22.665622 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce69d8fb05d3b4f877b142f86f624c3c.slice/crio-7784e964007ae3cca4b9daaae6c10680dda2bcf946d4c990d285b407d207b590 WatchSource:0}: Error finding container 7784e964007ae3cca4b9daaae6c10680dda2bcf946d4c990d285b407d207b590: Status 404 returned error can't find the container with id 7784e964007ae3cca4b9daaae6c10680dda2bcf946d4c990d285b407d207b590 Sep 09 14:33:22.703055 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.703034 2578 apiserver.go:52] "Watching apiserver" Sep 09 14:33:22.709419 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.709397 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 14:33:22.709774 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.709752 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-4v9k9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal","openshift-multus/multus-vsqcx","openshift-multus/network-metrics-daemon-fs9bf","openshift-ovn-kubernetes/ovnkube-node-66bb4","kube-system/global-pull-secret-syncer-k2jgj","kube-system/konnectivity-agent-rr629","openshift-cluster-node-tuning-operator/tuned-nf9q6","openshift-dns/node-resolver-qhwvw","openshift-multus/multus-additional-cni-plugins-rl6h9","openshift-network-diagnostics/network-check-target-5sm4r","openshift-network-operator/iptables-alerter-z5gj9","kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9"] Sep 09 14:33:22.712718 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.712698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.714856 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.714837 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:22.714952 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.714929 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:22.715221 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.715183 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 14:33:22.715281 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.715225 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 14:33:22.715281 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.715243 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 14:33:22.715395 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.715229 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 14:33:22.715395 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.715354 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jcmvs\"" Sep 09 14:33:22.715599 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.715585 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 14:33:22.717177 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.717159 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.717582 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.717557 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 14:33:22.721342 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.721042 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 14:33:22.721342 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.721073 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 14:33:22.721342 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.721088 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 14:33:22.721342 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.721113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.721342 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.721190 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5lfhj\"" Sep 09 14:33:22.723423 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.723402 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 14:33:22.723517 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.723490 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 14:33:22.723576 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.723519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.723654 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.723636 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xj9w8\"" Sep 09 14:33:22.725836 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.725813 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 14:33:22.725938 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.725816 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 14:33:22.725938 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.725906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.726110 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.726090 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 14:33:22.726315 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.726291 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 14:33:22.726421 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.726372 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fbvmh\"" Sep 09 14:33:22.726421 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.726392 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 14:33:22.726559 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.726543 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 14:33:22.728097 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.728071 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 14:33:22.728097 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.728072 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 14:33:22.728244 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.728231 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rwmxh\"" Sep 09 14:33:22.728319 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.728300 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 14:33:22.728395 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.728318 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.728461 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.728234 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 14:33:22.730566 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.730548 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 14:33:22.730660 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.730549 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-js2ts\"" Sep 09 14:33:22.730743 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.730729 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 14:33:22.730986 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.730969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:22.732350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af356e02-73e3-4111-a82b-e1462c92982b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.732459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af356e02-73e3-4111-a82b-e1462c92982b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.732459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-systemd\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.732459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af356e02-73e3-4111-a82b-e1462c92982b-cni-binary-copy\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.732459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-etc-selinux\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.732459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nj97\" (UniqueName: \"kubernetes.io/projected/f2fa5ef4-acc5-488f-9da4-1d8b94103656-kube-api-access-5nj97\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.732749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-sysconfig\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.732749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-var-lib-kubelet\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.732749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-run\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.732749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732585 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af356e02-73e3-4111-a82b-e1462c92982b-system-cni-dir\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.732749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-kubernetes\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.732749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.732749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-registration-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.732749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-device-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.732749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-sys-fs\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-host\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af356e02-73e3-4111-a82b-e1462c92982b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsrd\" (UniqueName: \"kubernetes.io/projected/af356e02-73e3-4111-a82b-e1462c92982b-kube-api-access-qrsrd\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-socket-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-sysctl-conf\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-tmp\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65vf9\" (UniqueName: \"kubernetes.io/projected/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-kube-api-access-65vf9\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.732976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-modprobe-d\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-sysctl-d\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-sys\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-tuned\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af356e02-73e3-4111-a82b-e1462c92982b-cnibin\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733125 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xqq5c\"" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-lib-modules\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733193 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 14:33:22.733258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733241 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 14:33:22.734018 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.733413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.735736 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.735717 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 14:33:22.735881 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.735738 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgl8q\"" Sep 09 14:33:22.735881 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.735818 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 14:33:22.735975 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.735717 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 14:33:22.736106 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.736088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:22.736191 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.736148 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:22.738701 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.738684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:22.738781 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.738740 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:22.741161 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.741144 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.743410 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.743354 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 14:33:22.743851 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.743828 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdlvt\"" Sep 09 14:33:22.754569 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.754544 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v4c9m" Sep 09 14:33:22.761130 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.761113 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v4c9m" Sep 09 14:33:22.818967 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.818946 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 14:33:22.833418 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b9fd951-66f4-459d-966d-6565d97cec02-host\") pod \"node-ca-4v9k9\" (UID: \"8b9fd951-66f4-459d-966d-6565d97cec02\") " pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.833503 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-slash\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.833503 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.833503 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fffaf1ff-bc62-4ce6-a5ca-a45d536a4217-hosts-file\") pod \"node-resolver-qhwvw\" (UID: \"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217\") " pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.833503 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-sysctl-d\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.833648 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-run-netns\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.833648 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-hostroot\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.833648 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-run-multus-certs\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.833648 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.833648 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-ovnkube-config\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.833648 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffaf1ff-bc62-4ce6-a5ca-a45d536a4217-tmp-dir\") pod \"node-resolver-qhwvw\" (UID: \"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217\") " pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.833828 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ddac112e-73f0-4043-a739-7dd8b01527aa-kubelet-config\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:22.833828 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:22.833828 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57zgx\" (UniqueName: \"kubernetes.io/projected/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-kube-api-access-57zgx\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.833828 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833694 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-sysctl-d\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.833828 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b67132ad-d647-4474-84cb-5991ab2ab883-agent-certs\") pod \"konnectivity-agent-rr629\" (UID: \"b67132ad-d647-4474-84cb-5991ab2ab883\") " pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:22.833828 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af356e02-73e3-4111-a82b-e1462c92982b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.833828 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-systemd\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.833828 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833782 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-var-lib-cni-multus\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.834132 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833836 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-systemd\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.834132 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-systemd-units\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.834132 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-ovn-node-metrics-cert\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.834132 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.833981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af356e02-73e3-4111-a82b-e1462c92982b-cni-binary-copy\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.834132 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nj97\" (UniqueName: \"kubernetes.io/projected/f2fa5ef4-acc5-488f-9da4-1d8b94103656-kube-api-access-5nj97\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.834132 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-sysconfig\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.834132 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-var-lib-kubelet\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.834350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-socket-dir-parent\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.834350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834154 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-var-lib-kubelet\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.834350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834181 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-sysconfig\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.834350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-run-k8s-cni-cncf-io\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.834350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-cni-netd\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.834350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-system-cni-dir\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.834350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-kubelet\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.834350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-env-overrides\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.834350 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-ovnkube-script-lib\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghpr\" (UniqueName: \"kubernetes.io/projected/fffaf1ff-bc62-4ce6-a5ca-a45d536a4217-kube-api-access-dghpr\") pod \"node-resolver-qhwvw\" (UID: \"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217\") " pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-kubernetes\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-var-lib-cni-bin\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4g9v\" (UniqueName: \"kubernetes.io/projected/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-kube-api-access-z4g9v\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-var-lib-openvswitch\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-kubernetes\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-device-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w559\" (UniqueName: \"kubernetes.io/projected/8b9fd951-66f4-459d-966d-6565d97cec02-kube-api-access-8w559\") pod \"node-ca-4v9k9\" (UID: \"8b9fd951-66f4-459d-966d-6565d97cec02\") " pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834629 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-cnibin\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.834653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834647 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-device-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af356e02-73e3-4111-a82b-e1462c92982b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsrd\" (UniqueName: \"kubernetes.io/projected/af356e02-73e3-4111-a82b-e1462c92982b-kube-api-access-qrsrd\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-log-socket\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-modprobe-d\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834819 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af356e02-73e3-4111-a82b-e1462c92982b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834879 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-modprobe-d\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-sys\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-tuned\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834940 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-sys\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-etc-openvswitch\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.834983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-run-openvswitch\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835006 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-node-log\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8pm\" (UniqueName: \"kubernetes.io/projected/b37cc860-a60b-44a7-b363-913eb3128dc7-kube-api-access-qm8pm\") pod \"iptables-alerter-z5gj9\" (UID: \"b37cc860-a60b-44a7-b363-913eb3128dc7\") " pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835079 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af356e02-73e3-4111-a82b-e1462c92982b-cnibin\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-lib-modules\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:22.835212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835149 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af356e02-73e3-4111-a82b-e1462c92982b-cnibin\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af356e02-73e3-4111-a82b-e1462c92982b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-cni-dir\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835195 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af356e02-73e3-4111-a82b-e1462c92982b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-daemon-config\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b67132ad-d647-4474-84cb-5991ab2ab883-konnectivity-ca\") pod \"konnectivity-agent-rr629\" (UID: \"b67132ad-d647-4474-84cb-5991ab2ab883\") " pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-etc-selinux\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-var-lib-kubelet\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835323 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-lib-modules\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-run\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-cni-binary-copy\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835382 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-etc-selinux\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835404 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-run\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-conf-dir\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-run-netns\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b37cc860-a60b-44a7-b363-913eb3128dc7-host-slash\") pod \"iptables-alerter-z5gj9\" (UID: \"b37cc860-a60b-44a7-b363-913eb3128dc7\") " pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.836029 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ddac112e-73f0-4043-a739-7dd8b01527aa-dbus\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af356e02-73e3-4111-a82b-e1462c92982b-system-cni-dir\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835540 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b9fd951-66f4-459d-966d-6565d97cec02-serviceca\") pod \"node-ca-4v9k9\" (UID: \"8b9fd951-66f4-459d-966d-6565d97cec02\") " pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-etc-kubernetes\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835574 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af356e02-73e3-4111-a82b-e1462c92982b-system-cni-dir\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-run-systemd\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835608 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-run-ovn\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b37cc860-a60b-44a7-b363-913eb3128dc7-iptables-alerter-script\") pod \"iptables-alerter-z5gj9\" (UID: \"b37cc860-a60b-44a7-b363-913eb3128dc7\") " pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835613 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af356e02-73e3-4111-a82b-e1462c92982b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835696 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-registration-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835713 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af356e02-73e3-4111-a82b-e1462c92982b-cni-binary-copy\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-sys-fs\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-host\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835788 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-registration-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l25gg\" (UniqueName: \"kubernetes.io/projected/c566cce4-7251-42bf-8696-f2505562d073-kube-api-access-l25gg\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835823 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-sys-fs\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.836668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835831 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-host\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.837181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-cni-bin\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.837181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-socket-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.837181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-sysctl-conf\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.837181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-tmp\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.837181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.835951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65vf9\" (UniqueName: \"kubernetes.io/projected/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-kube-api-access-65vf9\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.837181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.836017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f2fa5ef4-acc5-488f-9da4-1d8b94103656-socket-dir\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.837181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.836103 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-sysctl-conf\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.838888 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.838868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-tmp\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.838964 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.838892 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-etc-tuned\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.841061 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.841044 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:22.841106 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.841065 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:22.841106 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.841075 2578 projected.go:196] Error preparing data for projected volume kube-api-access-rnbbx for pod openshift-network-diagnostics/network-check-target-5sm4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:22.841181 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.841133 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx podName:e2f50ed1-8a0b-4bff-a86c-c46232240402 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:23.341118645 +0000 UTC m=+3.007924689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rnbbx" (UniqueName: "kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx") pod "network-check-target-5sm4r" (UID: "e2f50ed1-8a0b-4bff-a86c-c46232240402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:22.843544 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.843410 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nj97\" (UniqueName: \"kubernetes.io/projected/f2fa5ef4-acc5-488f-9da4-1d8b94103656-kube-api-access-5nj97\") pod \"aws-ebs-csi-driver-node-p87x9\" (UID: \"f2fa5ef4-acc5-488f-9da4-1d8b94103656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:22.843642 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.843467 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsrd\" (UniqueName: \"kubernetes.io/projected/af356e02-73e3-4111-a82b-e1462c92982b-kube-api-access-qrsrd\") pod \"multus-additional-cni-plugins-rl6h9\" (UID: \"af356e02-73e3-4111-a82b-e1462c92982b\") " pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:22.843703 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.843670 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65vf9\" (UniqueName: \"kubernetes.io/projected/54a4de0c-d5f5-40d8-bdfa-b90b1003a96c-kube-api-access-65vf9\") pod \"tuned-nf9q6\" (UID: \"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:22.936366 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-daemon-config\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.936539 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b67132ad-d647-4474-84cb-5991ab2ab883-konnectivity-ca\") pod \"konnectivity-agent-rr629\" (UID: \"b67132ad-d647-4474-84cb-5991ab2ab883\") " pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:22.936539 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-var-lib-kubelet\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.936655 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:22.936655 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-cni-binary-copy\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.936655 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936634 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-var-lib-kubelet\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.936813 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-conf-dir\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.936813 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.936701 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:22.936813 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-run-netns\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.936813 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b37cc860-a60b-44a7-b363-913eb3128dc7-host-slash\") pod \"iptables-alerter-z5gj9\" (UID: \"b37cc860-a60b-44a7-b363-913eb3128dc7\") " pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.936813 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ddac112e-73f0-4043-a739-7dd8b01527aa-dbus\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:22.936813 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.936799 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret podName:ddac112e-73f0-4043-a739-7dd8b01527aa nodeName:}" failed. No retries permitted until 2026-09-09 14:33:23.436778567 +0000 UTC m=+3.103584616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret") pod "global-pull-secret-syncer-k2jgj" (UID: "ddac112e-73f0-4043-a739-7dd8b01527aa") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936811 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-conf-dir\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b9fd951-66f4-459d-966d-6565d97cec02-serviceca\") pod \"node-ca-4v9k9\" (UID: \"8b9fd951-66f4-459d-966d-6565d97cec02\") " pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-etc-kubernetes\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-run-systemd\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936912 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b37cc860-a60b-44a7-b363-913eb3128dc7-host-slash\") pod \"iptables-alerter-z5gj9\" (UID: \"b37cc860-a60b-44a7-b363-913eb3128dc7\") " pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936919 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ddac112e-73f0-4043-a739-7dd8b01527aa-dbus\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936862 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-run-netns\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-run-ovn\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936950 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-run-ovn\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.936993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b37cc860-a60b-44a7-b363-913eb3128dc7-iptables-alerter-script\") pod \"iptables-alerter-z5gj9\" (UID: \"b37cc860-a60b-44a7-b363-913eb3128dc7\") " pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937006 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-etc-kubernetes\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l25gg\" (UniqueName: \"kubernetes.io/projected/c566cce4-7251-42bf-8696-f2505562d073-kube-api-access-l25gg\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-cni-bin\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-run-systemd\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b9fd951-66f4-459d-966d-6565d97cec02-host\") pod \"node-ca-4v9k9\" (UID: \"8b9fd951-66f4-459d-966d-6565d97cec02\") " pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-slash\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937113 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-cni-bin\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937057 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-daemon-config\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b67132ad-d647-4474-84cb-5991ab2ab883-konnectivity-ca\") pod \"konnectivity-agent-rr629\" (UID: \"b67132ad-d647-4474-84cb-5991ab2ab883\") " pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937163 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b9fd951-66f4-459d-966d-6565d97cec02-host\") pod \"node-ca-4v9k9\" (UID: \"8b9fd951-66f4-459d-966d-6565d97cec02\") " pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fffaf1ff-bc62-4ce6-a5ca-a45d536a4217-hosts-file\") pod \"node-resolver-qhwvw\" (UID: \"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217\") " pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-run-netns\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937263 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937255 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-slash\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-hostroot\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937309 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-hostroot\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937314 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fffaf1ff-bc62-4ce6-a5ca-a45d536a4217-hosts-file\") pod \"node-resolver-qhwvw\" (UID: \"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217\") " pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-run-multus-certs\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937351 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-run-netns\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-ovnkube-config\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937402 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b9fd951-66f4-459d-966d-6565d97cec02-serviceca\") pod \"node-ca-4v9k9\" (UID: \"8b9fd951-66f4-459d-966d-6565d97cec02\") " pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffaf1ff-bc62-4ce6-a5ca-a45d536a4217-tmp-dir\") pod \"node-resolver-qhwvw\" (UID: \"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217\") " pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.937979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ddac112e-73f0-4043-a739-7dd8b01527aa-kubelet-config\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57zgx\" (UniqueName: \"kubernetes.io/projected/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-kube-api-access-57zgx\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937412 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b67132ad-d647-4474-84cb-5991ab2ab883-agent-certs\") pod \"konnectivity-agent-rr629\" (UID: \"b67132ad-d647-4474-84cb-5991ab2ab883\") " pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937392 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-run-multus-certs\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937588 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ddac112e-73f0-4043-a739-7dd8b01527aa-kubelet-config\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-var-lib-cni-multus\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-systemd-units\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937642 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b37cc860-a60b-44a7-b363-913eb3128dc7-iptables-alerter-script\") pod \"iptables-alerter-z5gj9\" (UID: \"b37cc860-a60b-44a7-b363-913eb3128dc7\") " pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-ovn-node-metrics-cert\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937664 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-var-lib-cni-multus\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-socket-dir-parent\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-run-k8s-cni-cncf-io\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937725 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-cni-binary-copy\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937702 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-systemd-units\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-cni-netd\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937799 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-run-k8s-cni-cncf-io\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.938864 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937801 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-socket-dir-parent\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-system-cni-dir\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937839 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffaf1ff-bc62-4ce6-a5ca-a45d536a4217-tmp-dir\") pod \"node-resolver-qhwvw\" (UID: \"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217\") " pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-kubelet\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937864 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-cni-netd\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-env-overrides\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937897 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-system-cni-dir\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937907 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-host-kubelet\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-ovnkube-script-lib\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghpr\" (UniqueName: \"kubernetes.io/projected/fffaf1ff-bc62-4ce6-a5ca-a45d536a4217-kube-api-access-dghpr\") pod \"node-resolver-qhwvw\" (UID: \"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217\") " pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.937977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-var-lib-cni-bin\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4g9v\" (UniqueName: \"kubernetes.io/projected/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-kube-api-access-z4g9v\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938043 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-host-var-lib-cni-bin\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-var-lib-openvswitch\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w559\" (UniqueName: \"kubernetes.io/projected/8b9fd951-66f4-459d-966d-6565d97cec02-kube-api-access-8w559\") pod \"node-ca-4v9k9\" (UID: \"8b9fd951-66f4-459d-966d-6565d97cec02\") " pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-cnibin\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-log-socket\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-var-lib-openvswitch\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.939411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-etc-openvswitch\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-run-openvswitch\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-node-log\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8pm\" (UniqueName: \"kubernetes.io/projected/b37cc860-a60b-44a7-b363-913eb3128dc7-kube-api-access-qm8pm\") pod \"iptables-alerter-z5gj9\" (UID: \"b37cc860-a60b-44a7-b363-913eb3128dc7\") " pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938393 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-etc-openvswitch\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938426 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-ovnkube-config\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938437 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-node-log\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938351 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-env-overrides\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938458 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-ovnkube-script-lib\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938486 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-run-openvswitch\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938505 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-cnibin\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.938516 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938516 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-log-socket\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:22.938578 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:23.438555706 +0000 UTC m=+3.105361737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-cni-dir\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.940015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.938718 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-multus-cni-dir\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.940496 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.940368 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-ovn-node-metrics-cert\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.940532 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.940523 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b67132ad-d647-4474-84cb-5991ab2ab883-agent-certs\") pod \"konnectivity-agent-rr629\" (UID: \"b67132ad-d647-4474-84cb-5991ab2ab883\") " pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:22.948988 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.948961 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l25gg\" (UniqueName: \"kubernetes.io/projected/c566cce4-7251-42bf-8696-f2505562d073-kube-api-access-l25gg\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:22.949321 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.949302 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8pm\" (UniqueName: \"kubernetes.io/projected/b37cc860-a60b-44a7-b363-913eb3128dc7-kube-api-access-qm8pm\") pod \"iptables-alerter-z5gj9\" (UID: \"b37cc860-a60b-44a7-b363-913eb3128dc7\") " pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:22.949541 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.949524 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w559\" (UniqueName: \"kubernetes.io/projected/8b9fd951-66f4-459d-966d-6565d97cec02-kube-api-access-8w559\") pod \"node-ca-4v9k9\" (UID: \"8b9fd951-66f4-459d-966d-6565d97cec02\") " pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:22.949851 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.949835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57zgx\" (UniqueName: \"kubernetes.io/projected/bfe14675-8e24-4434-a00e-e9fb8f83e0a6-kube-api-access-57zgx\") pod \"ovnkube-node-66bb4\" (UID: \"bfe14675-8e24-4434-a00e-e9fb8f83e0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:22.950177 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.950158 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4g9v\" (UniqueName: \"kubernetes.io/projected/e8fcd125-d61b-4c71-a1a2-b7bb507101b1-kube-api-access-z4g9v\") pod \"multus-vsqcx\" (UID: \"e8fcd125-d61b-4c71-a1a2-b7bb507101b1\") " pod="openshift-multus/multus-vsqcx" Sep 09 14:33:22.950177 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.950172 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghpr\" (UniqueName: \"kubernetes.io/projected/fffaf1ff-bc62-4ce6-a5ca-a45d536a4217-kube-api-access-dghpr\") pod \"node-resolver-qhwvw\" (UID: \"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217\") " pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:22.982569 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.982526 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal" event={"ID":"ce69d8fb05d3b4f877b142f86f624c3c","Type":"ContainerStarted","Data":"7784e964007ae3cca4b9daaae6c10680dda2bcf946d4c990d285b407d207b590"} Sep 09 14:33:22.983689 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:22.983670 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" event={"ID":"dfc2c7a82d8ba99d752d94210c868394","Type":"ContainerStarted","Data":"455ff97ecd6556557e57106c959f19b23fbbab5cbd56de4c3d256621d6fc3550"} Sep 09 14:33:23.026239 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.026189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" Sep 09 14:33:23.033853 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.033834 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" Sep 09 14:33:23.039246 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:23.039220 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf356e02_73e3_4111_a82b_e1462c92982b.slice/crio-be83054fb4ce95f71bc507961f8c61f82295895516ef5ead83956769c8b7e046 WatchSource:0}: Error finding container be83054fb4ce95f71bc507961f8c61f82295895516ef5ead83956769c8b7e046: Status 404 returned error can't find the container with id be83054fb4ce95f71bc507961f8c61f82295895516ef5ead83956769c8b7e046 Sep 09 14:33:23.041317 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.041298 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" Sep 09 14:33:23.043613 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:23.043588 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fa5ef4_acc5_488f_9da4_1d8b94103656.slice/crio-2d7610925919ebc00b9c0a95bc397db7c1d1be34b013370c4f861fe97f150e8a WatchSource:0}: Error finding container 2d7610925919ebc00b9c0a95bc397db7c1d1be34b013370c4f861fe97f150e8a: Status 404 returned error can't find the container with id 2d7610925919ebc00b9c0a95bc397db7c1d1be34b013370c4f861fe97f150e8a Sep 09 14:33:23.047204 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.047105 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:23.053356 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.053334 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z5gj9" Sep 09 14:33:23.053734 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:23.053709 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a4de0c_d5f5_40d8_bdfa_b90b1003a96c.slice/crio-953138649e90d31dbf3ba4f9206a4f1948f033ecc56cee7525e3ecb5eb5f77a6 WatchSource:0}: Error finding container 953138649e90d31dbf3ba4f9206a4f1948f033ecc56cee7525e3ecb5eb5f77a6: Status 404 returned error can't find the container with id 953138649e90d31dbf3ba4f9206a4f1948f033ecc56cee7525e3ecb5eb5f77a6 Sep 09 14:33:23.059520 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:23.059469 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe14675_8e24_4434_a00e_e9fb8f83e0a6.slice/crio-ae748770a78e1c534728567aa501c2d9e994a7d012c042cae9e42a801bb31e96 WatchSource:0}: Error finding container ae748770a78e1c534728567aa501c2d9e994a7d012c042cae9e42a801bb31e96: Status 404 returned error can't find the container with id ae748770a78e1c534728567aa501c2d9e994a7d012c042cae9e42a801bb31e96 Sep 09 14:33:23.060231 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.060206 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qhwvw" Sep 09 14:33:23.063581 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:23.063558 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37cc860_a60b_44a7_b363_913eb3128dc7.slice/crio-39d90db11ac4dc0ba989de5604d52b36212bd6da08f501a9ec4d61b5aa98e62e WatchSource:0}: Error finding container 39d90db11ac4dc0ba989de5604d52b36212bd6da08f501a9ec4d61b5aa98e62e: Status 404 returned error can't find the container with id 39d90db11ac4dc0ba989de5604d52b36212bd6da08f501a9ec4d61b5aa98e62e Sep 09 14:33:23.064759 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.064735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:23.070608 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.070584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4v9k9" Sep 09 14:33:23.070929 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:23.070907 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffaf1ff_bc62_4ce6_a5ca_a45d536a4217.slice/crio-c4934eaee37b7f946d3902e296685d225a1bda212e714ed6580a4c5ff882f10e WatchSource:0}: Error finding container c4934eaee37b7f946d3902e296685d225a1bda212e714ed6580a4c5ff882f10e: Status 404 returned error can't find the container with id c4934eaee37b7f946d3902e296685d225a1bda212e714ed6580a4c5ff882f10e Sep 09 14:33:23.075280 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.075260 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vsqcx" Sep 09 14:33:23.075513 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:23.075494 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67132ad_d647_4474_84cb_5991ab2ab883.slice/crio-ad7de737d6dd1ac2c6dd66b2b3f08d47cef24e55c7fbee392ed5a8d2d6cc1b88 WatchSource:0}: Error finding container ad7de737d6dd1ac2c6dd66b2b3f08d47cef24e55c7fbee392ed5a8d2d6cc1b88: Status 404 returned error can't find the container with id ad7de737d6dd1ac2c6dd66b2b3f08d47cef24e55c7fbee392ed5a8d2d6cc1b88 Sep 09 14:33:23.080425 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:23.080385 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9fd951_66f4_459d_966d_6565d97cec02.slice/crio-3f28a877a86aea182ee872e023d5fec1a3789374d893751b72ed41cef19b71ac WatchSource:0}: Error finding container 3f28a877a86aea182ee872e023d5fec1a3789374d893751b72ed41cef19b71ac: Status 404 returned error can't find the container with id 3f28a877a86aea182ee872e023d5fec1a3789374d893751b72ed41cef19b71ac Sep 09 14:33:23.084697 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:23.084677 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fcd125_d61b_4c71_a1a2_b7bb507101b1.slice/crio-6a2c0a39f404c17ae9b87145bc72e0b2a1812819920c4fbd100a1447e34237a5 WatchSource:0}: Error finding container 6a2c0a39f404c17ae9b87145bc72e0b2a1812819920c4fbd100a1447e34237a5: Status 404 returned error can't find the container with id 6a2c0a39f404c17ae9b87145bc72e0b2a1812819920c4fbd100a1447e34237a5 Sep 09 14:33:23.341265 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.341166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:23.341452 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:23.341331 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:23.341452 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:23.341354 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:23.341452 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:23.341367 2578 projected.go:196] Error preparing data for projected volume kube-api-access-rnbbx for pod openshift-network-diagnostics/network-check-target-5sm4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:23.341452 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:23.341448 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx podName:e2f50ed1-8a0b-4bff-a86c-c46232240402 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:24.341427794 +0000 UTC m=+4.008233819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnbbx" (UniqueName: "kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx") pod "network-check-target-5sm4r" (UID: "e2f50ed1-8a0b-4bff-a86c-c46232240402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:23.441938 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.441904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:23.441938 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.441947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:23.442149 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:23.442063 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:23.442149 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:23.442116 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret podName:ddac112e-73f0-4043-a739-7dd8b01527aa nodeName:}" failed. No retries permitted until 2026-09-09 14:33:24.442101383 +0000 UTC m=+4.108907409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret") pod "global-pull-secret-syncer-k2jgj" (UID: "ddac112e-73f0-4043-a739-7dd8b01527aa") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:23.442480 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:23.442459 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:23.442620 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:23.442519 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:24.442500323 +0000 UTC m=+4.109306352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:23.766535 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.766362 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 14:28:23 +0000 UTC" deadline="2028-03-09 23:23:13.821766947 +0000 UTC" Sep 09 14:33:23.766535 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.766441 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13136h49m50.055330383s" Sep 09 14:33:23.976281 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.976251 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:23.976488 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:23.976387 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:23.991776 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.991735 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" event={"ID":"f2fa5ef4-acc5-488f-9da4-1d8b94103656","Type":"ContainerStarted","Data":"2d7610925919ebc00b9c0a95bc397db7c1d1be34b013370c4f861fe97f150e8a"} Sep 09 14:33:23.995968 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:23.995938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" event={"ID":"af356e02-73e3-4111-a82b-e1462c92982b","Type":"ContainerStarted","Data":"be83054fb4ce95f71bc507961f8c61f82295895516ef5ead83956769c8b7e046"} Sep 09 14:33:24.002589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.002561 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rr629" event={"ID":"b67132ad-d647-4474-84cb-5991ab2ab883","Type":"ContainerStarted","Data":"ad7de737d6dd1ac2c6dd66b2b3f08d47cef24e55c7fbee392ed5a8d2d6cc1b88"} Sep 09 14:33:24.007832 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.007804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z5gj9" event={"ID":"b37cc860-a60b-44a7-b363-913eb3128dc7","Type":"ContainerStarted","Data":"39d90db11ac4dc0ba989de5604d52b36212bd6da08f501a9ec4d61b5aa98e62e"} Sep 09 14:33:24.014984 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.014959 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" event={"ID":"bfe14675-8e24-4434-a00e-e9fb8f83e0a6","Type":"ContainerStarted","Data":"ae748770a78e1c534728567aa501c2d9e994a7d012c042cae9e42a801bb31e96"} Sep 09 14:33:24.021634 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.021530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" event={"ID":"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c","Type":"ContainerStarted","Data":"953138649e90d31dbf3ba4f9206a4f1948f033ecc56cee7525e3ecb5eb5f77a6"} Sep 09 14:33:24.044990 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.044951 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vsqcx" event={"ID":"e8fcd125-d61b-4c71-a1a2-b7bb507101b1","Type":"ContainerStarted","Data":"6a2c0a39f404c17ae9b87145bc72e0b2a1812819920c4fbd100a1447e34237a5"} Sep 09 14:33:24.051584 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.051553 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4v9k9" event={"ID":"8b9fd951-66f4-459d-966d-6565d97cec02","Type":"ContainerStarted","Data":"3f28a877a86aea182ee872e023d5fec1a3789374d893751b72ed41cef19b71ac"} Sep 09 14:33:24.054247 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.054217 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qhwvw" event={"ID":"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217","Type":"ContainerStarted","Data":"c4934eaee37b7f946d3902e296685d225a1bda212e714ed6580a4c5ff882f10e"} Sep 09 14:33:24.351964 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.351876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:24.352139 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.352054 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:24.352139 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.352075 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:24.352139 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.352089 2578 projected.go:196] Error preparing data for projected volume kube-api-access-rnbbx for pod openshift-network-diagnostics/network-check-target-5sm4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:24.352314 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.352156 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx podName:e2f50ed1-8a0b-4bff-a86c-c46232240402 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:26.352136598 +0000 UTC m=+6.018942639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnbbx" (UniqueName: "kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx") pod "network-check-target-5sm4r" (UID: "e2f50ed1-8a0b-4bff-a86c-c46232240402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:24.378491 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.378459 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 14:33:24.453137 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.453104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:24.453308 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.453154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:24.453397 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.453308 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:24.453397 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.453369 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret podName:ddac112e-73f0-4043-a739-7dd8b01527aa nodeName:}" failed. No retries permitted until 2026-09-09 14:33:26.453349892 +0000 UTC m=+6.120155920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret") pod "global-pull-secret-syncer-k2jgj" (UID: "ddac112e-73f0-4043-a739-7dd8b01527aa") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:24.453803 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.453783 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:24.453869 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.453837 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:26.453822806 +0000 UTC m=+6.120628839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:24.767088 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.766989 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 14:28:23 +0000 UTC" deadline="2028-06-05 07:05:25.220519552 +0000 UTC" Sep 09 14:33:24.767088 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.767040 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15232h32m0.453485328s" Sep 09 14:33:24.978094 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.978058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:24.978285 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.978199 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:24.978741 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:24.978719 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:24.978910 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:24.978842 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:25.978110 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:25.977540 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:25.978110 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:25.977681 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:26.369646 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:26.369382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:26.369646 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.369619 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:26.369646 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.369642 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:26.369646 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.369655 2578 projected.go:196] Error preparing data for projected volume kube-api-access-rnbbx for pod openshift-network-diagnostics/network-check-target-5sm4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:26.369926 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.369726 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx podName:e2f50ed1-8a0b-4bff-a86c-c46232240402 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:30.369705979 +0000 UTC m=+10.036512029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnbbx" (UniqueName: "kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx") pod "network-check-target-5sm4r" (UID: "e2f50ed1-8a0b-4bff-a86c-c46232240402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:26.470037 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:26.469991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:26.470214 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:26.470100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:26.470281 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.470237 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:26.470324 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.470303 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:30.470283379 +0000 UTC m=+10.137089410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:26.470771 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.470742 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:26.470896 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.470812 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret podName:ddac112e-73f0-4043-a739-7dd8b01527aa nodeName:}" failed. No retries permitted until 2026-09-09 14:33:30.470795659 +0000 UTC m=+10.137601691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret") pod "global-pull-secret-syncer-k2jgj" (UID: "ddac112e-73f0-4043-a739-7dd8b01527aa") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:26.979363 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:26.978681 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:26.979363 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:26.978735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:26.979363 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.978887 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:26.979363 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:26.979281 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:27.976773 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:27.976743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:27.976952 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:27.976857 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:28.976664 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:28.976542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:28.977158 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:28.976696 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:28.977158 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:28.976719 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:28.977158 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:28.976838 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:29.977103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:29.977033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:29.977653 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:29.977229 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:30.405739 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:30.405627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:30.405918 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.405899 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:30.405974 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.405925 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:30.405974 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.405939 2578 projected.go:196] Error preparing data for projected volume kube-api-access-rnbbx for pod openshift-network-diagnostics/network-check-target-5sm4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:30.406053 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.406029 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx podName:e2f50ed1-8a0b-4bff-a86c-c46232240402 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:38.406009116 +0000 UTC m=+18.072815141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnbbx" (UniqueName: "kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx") pod "network-check-target-5sm4r" (UID: "e2f50ed1-8a0b-4bff-a86c-c46232240402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:30.506344 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:30.506300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:30.506518 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:30.506364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:30.506518 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.506471 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:30.506518 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.506491 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:30.506752 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.506549 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:38.506526958 +0000 UTC m=+18.173332988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:30.506752 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.506571 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret podName:ddac112e-73f0-4043-a739-7dd8b01527aa nodeName:}" failed. No retries permitted until 2026-09-09 14:33:38.506560054 +0000 UTC m=+18.173366081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret") pod "global-pull-secret-syncer-k2jgj" (UID: "ddac112e-73f0-4043-a739-7dd8b01527aa") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:30.977871 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:30.977341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:30.977871 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.977497 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:30.977871 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:30.977780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:30.978515 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:30.977897 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:31.976129 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:31.976090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:31.976296 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:31.976226 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:32.977489 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:32.977455 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:32.977930 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:32.977596 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:32.977992 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:32.977931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:32.978056 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:32.978019 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:33.976652 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:33.976627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:33.976765 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:33.976746 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:34.976728 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:34.976667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:34.977314 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:34.976805 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:34.977314 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:34.977017 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:34.977314 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:34.977149 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:35.976137 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:35.976101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:35.976321 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:35.976224 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:36.976815 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:36.976736 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:36.976815 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:36.976774 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:36.977322 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:36.976881 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:36.977322 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:36.977011 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:37.977022 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:37.976997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:37.977367 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:37.977093 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:38.463882 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:38.463799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:38.464131 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.463978 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:38.464131 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.463999 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:38.464131 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.464013 2578 projected.go:196] Error preparing data for projected volume kube-api-access-rnbbx for pod openshift-network-diagnostics/network-check-target-5sm4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:38.464131 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.464081 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx podName:e2f50ed1-8a0b-4bff-a86c-c46232240402 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:54.464062919 +0000 UTC m=+34.130868949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnbbx" (UniqueName: "kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx") pod "network-check-target-5sm4r" (UID: "e2f50ed1-8a0b-4bff-a86c-c46232240402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:38.564244 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:38.564203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:38.564244 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:38.564249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:38.564491 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.564395 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:38.564491 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.564465 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret podName:ddac112e-73f0-4043-a739-7dd8b01527aa nodeName:}" failed. No retries permitted until 2026-09-09 14:33:54.564449524 +0000 UTC m=+34.231255547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret") pod "global-pull-secret-syncer-k2jgj" (UID: "ddac112e-73f0-4043-a739-7dd8b01527aa") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:38.564601 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.564397 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:38.564601 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.564552 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:54.564534358 +0000 UTC m=+34.231340398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:38.976522 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:38.976486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:38.976704 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.976630 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:38.976775 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:38.976720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:38.976833 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:38.976816 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:39.976751 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:39.976715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:39.977214 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:39.976858 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:40.977364 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:40.977322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:40.977840 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:40.977447 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:40.977840 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:40.977510 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:40.977840 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:40.977623 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:41.976247 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:41.976024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:41.976442 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:41.976404 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:42.100192 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.100142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal" event={"ID":"ce69d8fb05d3b4f877b142f86f624c3c","Type":"ContainerStarted","Data":"7395211f5b7c8369ecafd36efb306baa3c4b8ebc39e949812cbd33d7a233b929"} Sep 09 14:33:42.105221 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.104991 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" event={"ID":"bfe14675-8e24-4434-a00e-e9fb8f83e0a6","Type":"ContainerStarted","Data":"5d20ede8ee57b80308816d2c197216ef2b82481137737ef0a26836f57de36438"} Sep 09 14:33:42.105546 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.105270 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" event={"ID":"bfe14675-8e24-4434-a00e-e9fb8f83e0a6","Type":"ContainerStarted","Data":"ad967447de83a800cbf663fbba0f4642c643293057ecb45a2c3ee2a3b7b5a38b"} Sep 09 14:33:42.113266 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.113231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" event={"ID":"54a4de0c-d5f5-40d8-bdfa-b90b1003a96c","Type":"ContainerStarted","Data":"1b7cec92c78ace0e4ced301b928055d1d439ce381ca2aee808d80a7384f5247e"} Sep 09 14:33:42.115710 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.115684 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vsqcx" event={"ID":"e8fcd125-d61b-4c71-a1a2-b7bb507101b1","Type":"ContainerStarted","Data":"145d5eb7bfd3bf7d8cc0d6f6551c0b46a3b7c8ddbb7089a8320e093412f432ea"} Sep 09 14:33:42.120025 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.119894 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-173.ec2.internal" podStartSLOduration=20.119862428 podStartE2EDuration="20.119862428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:42.118493159 +0000 UTC m=+21.785299220" watchObservedRunningTime="2026-09-09 14:33:42.119862428 +0000 UTC m=+21.786668475" Sep 09 14:33:42.140083 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.140042 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nf9q6" podStartSLOduration=2.41451186 podStartE2EDuration="21.140028486s" totalImagesPullingTime="18.725516626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:42.139806832 +0000 UTC m=+21.806612889" watchObservedRunningTime="2026-09-09 14:33:42.140028486 +0000 UTC m=+21.806834532" Sep 09 14:33:42.162831 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.162774 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vsqcx" podStartSLOduration=1.386909194 podStartE2EDuration="20.162752797s" totalImagesPullingTime="18.775843603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:42.162529233 +0000 UTC m=+21.829335282" watchObservedRunningTime="2026-09-09 14:33:42.162752797 +0000 UTC m=+21.829558848" Sep 09 14:33:42.976161 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.976131 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:42.976294 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:42.976270 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:42.976706 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:42.976684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:42.976812 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:42.976761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:43.023451 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.023431 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 14:33:43.119371 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.119280 2578 generic.go:356] "Generic (PLEG): container finished" podID="af356e02-73e3-4111-a82b-e1462c92982b" containerID="dfe6ee10bdb0848e51663e28a4547be9b5d1ce6ec7d66e17699616d44e192aac" exitCode=0 Sep 09 14:33:43.120112 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.119382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" event={"ID":"af356e02-73e3-4111-a82b-e1462c92982b","Type":"ContainerDied","Data":"dfe6ee10bdb0848e51663e28a4547be9b5d1ce6ec7d66e17699616d44e192aac"} Sep 09 14:33:43.121064 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.121041 2578 generic.go:356] "Generic (PLEG): container finished" podID="dfc2c7a82d8ba99d752d94210c868394" containerID="ca92c2c0cca738ad0f90dc10d1ba0b52f306343ce2cab41c1e56649b8b31a4a3" exitCode=0 Sep 09 14:33:43.121151 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.121114 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" event={"ID":"dfc2c7a82d8ba99d752d94210c868394","Type":"ContainerDied","Data":"ca92c2c0cca738ad0f90dc10d1ba0b52f306343ce2cab41c1e56649b8b31a4a3"} Sep 09 14:33:43.122603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.122572 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rr629" event={"ID":"b67132ad-d647-4474-84cb-5991ab2ab883","Type":"ContainerStarted","Data":"5a62859d8712f728269e11e1f98cef1ac9edd30d4dc0265c0c82317592eb6f6e"} Sep 09 14:33:43.124009 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.123990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z5gj9" event={"ID":"b37cc860-a60b-44a7-b363-913eb3128dc7","Type":"ContainerStarted","Data":"1f06eca34ea68f2e24ef5eabf9fe95b5658e2bfa0e2d9a9ca4921590a5729c12"} Sep 09 14:33:43.130075 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.127692 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" event={"ID":"bfe14675-8e24-4434-a00e-e9fb8f83e0a6","Type":"ContainerStarted","Data":"0698f21e63e3b6075618936667eaf47094bdd71efd0c5fe29fe3dea9d98f65ef"} Sep 09 14:33:43.130075 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.127823 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" event={"ID":"bfe14675-8e24-4434-a00e-e9fb8f83e0a6","Type":"ContainerStarted","Data":"1683e13d98ea2b4ac8163b49b98ac39a844b9d61d0489fc02b4052fca5fc27c8"} Sep 09 14:33:43.130075 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.127890 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" event={"ID":"bfe14675-8e24-4434-a00e-e9fb8f83e0a6","Type":"ContainerStarted","Data":"372175b581b24546da31d9045ed4ca0855bc008a6a92d4eb6dbe52221b0db605"} Sep 09 14:33:43.130075 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.127957 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" event={"ID":"bfe14675-8e24-4434-a00e-e9fb8f83e0a6","Type":"ContainerStarted","Data":"f2b75ca26d0370a338392187dd6f3be07fab8b7218f8c96d6c3d6a4a22eaa7c1"} Sep 09 14:33:43.130654 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.130620 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4v9k9" event={"ID":"8b9fd951-66f4-459d-966d-6565d97cec02","Type":"ContainerStarted","Data":"29258f67b61463a4091be97ad07bfa66e413454aa6182aa5fb87f39dc96d5635"} Sep 09 14:33:43.132026 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.132006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qhwvw" event={"ID":"fffaf1ff-bc62-4ce6-a5ca-a45d536a4217","Type":"ContainerStarted","Data":"7cff002e9e1e72ae64e93d674129462d5784ba86b7827c2a74dd0313bc8dd76a"} Sep 09 14:33:43.133672 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.133650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" event={"ID":"f2fa5ef4-acc5-488f-9da4-1d8b94103656","Type":"ContainerStarted","Data":"2566de7b1bdae277c2c7bc24d9b1653a4bda211fa650295c30b925585a502e8f"} Sep 09 14:33:43.133753 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.133700 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" event={"ID":"f2fa5ef4-acc5-488f-9da4-1d8b94103656","Type":"ContainerStarted","Data":"f764df4c9ced1e9213fd73ba1ba879a27b41b2655a1447fdcf4b41e30f908bc8"} Sep 09 14:33:43.157168 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.157130 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4v9k9" podStartSLOduration=3.4621835020000002 podStartE2EDuration="22.157114862s" totalImagesPullingTime="18.69493136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:43.156657555 +0000 UTC m=+22.823463603" watchObservedRunningTime="2026-09-09 14:33:43.157114862 +0000 UTC m=+22.823920910" Sep 09 14:33:43.174156 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.174112 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z5gj9" podStartSLOduration=3.458597105 podStartE2EDuration="22.174100708s" totalImagesPullingTime="18.715503603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:43.174059078 +0000 UTC m=+22.840865137" watchObservedRunningTime="2026-09-09 14:33:43.174100708 +0000 UTC m=+22.840906787" Sep 09 14:33:43.196787 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.196752 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qhwvw" podStartSLOduration=3.492654942 podStartE2EDuration="22.19674074s" totalImagesPullingTime="18.704085798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:43.196357967 +0000 UTC m=+22.863164007" watchObservedRunningTime="2026-09-09 14:33:43.19674074 +0000 UTC m=+22.863546785" Sep 09 14:33:43.237795 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.237626 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rr629" podStartSLOduration=11.024551309 podStartE2EDuration="22.237611946s" totalImagesPullingTime="11.213060637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:43.237428949 +0000 UTC m=+22.904234995" watchObservedRunningTime="2026-09-09 14:33:43.237611946 +0000 UTC m=+22.904417992" Sep 09 14:33:43.789533 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.789402 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T14:33:43.023447817Z","UUID":"a4010230-5fef-4a53-b04c-4e8e9c9a84ee","Handler":null,"Name":"","Endpoint":""} Sep 09 14:33:43.791257 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.791231 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 14:33:43.791415 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.791273 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 14:33:43.976975 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:43.976949 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:43.977098 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:43.977076 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:44.138194 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:44.138092 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" event={"ID":"dfc2c7a82d8ba99d752d94210c868394","Type":"ContainerStarted","Data":"8adbc3b2ab96a41e632c46a6facf8758fe45af7e9586a9eb67c324614bd4d63b"} Sep 09 14:33:44.140636 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:44.140609 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" event={"ID":"f2fa5ef4-acc5-488f-9da4-1d8b94103656","Type":"ContainerStarted","Data":"cc8bba1ee45a30f7d71a779028f593a414769688e974d360e29a13ea0ec39db9"} Sep 09 14:33:44.153953 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:44.153909 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-173.ec2.internal" podStartSLOduration=22.153891921 podStartE2EDuration="22.153891921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:44.152950647 +0000 UTC m=+23.819756692" watchObservedRunningTime="2026-09-09 14:33:44.153891921 +0000 UTC m=+23.820697966" Sep 09 14:33:44.576697 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:44.576657 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:44.577294 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:44.577264 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:44.592722 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:44.592677 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p87x9" podStartSLOduration=3.27210575 podStartE2EDuration="23.592657349s" totalImagesPullingTime="20.320551599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:44.170778108 +0000 UTC m=+23.837584154" watchObservedRunningTime="2026-09-09 14:33:44.592657349 +0000 UTC m=+24.259463397" Sep 09 14:33:44.977126 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:44.977093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:44.977308 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:44.977098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:44.977308 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:44.977220 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:44.977427 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:44.977300 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:45.148532 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:45.148486 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" event={"ID":"bfe14675-8e24-4434-a00e-e9fb8f83e0a6","Type":"ContainerStarted","Data":"fc7da0c3a6bb76a4ffa0c24c41b1e3c4ee8c29492480c20c60aec76376a9c7d2"} Sep 09 14:33:45.976525 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:45.976490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:45.976683 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:45.976621 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:46.150761 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:46.150725 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 14:33:46.976160 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:46.976135 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:46.976262 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:46.976179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:46.976323 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:46.976273 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:46.976396 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:46.976363 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:47.156623 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:47.156590 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" event={"ID":"bfe14675-8e24-4434-a00e-e9fb8f83e0a6","Type":"ContainerStarted","Data":"8c8ab4f83d1cccfcb7b98a8c40159168ecbb61c28f540b6f5a2126da62169986"} Sep 09 14:33:47.157015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:47.156790 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:47.157015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:47.156860 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:47.157015 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:47.156911 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:47.176155 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:47.175944 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:47.176299 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:47.176279 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:33:47.183395 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:47.183339 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" podStartSLOduration=7.414085696 podStartE2EDuration="26.183324364s" totalImagesPullingTime="18.769238668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:47.182221542 +0000 UTC m=+26.849027590" watchObservedRunningTime="2026-09-09 14:33:47.183324364 +0000 UTC m=+26.850130405" Sep 09 14:33:47.976569 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:47.976543 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:47.976756 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:47.976631 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:48.061383 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.061336 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:48.061551 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.061467 2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 14:33:48.061891 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.061873 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rr629" Sep 09 14:33:48.160462 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.160428 2578 generic.go:356] "Generic (PLEG): container finished" podID="af356e02-73e3-4111-a82b-e1462c92982b" containerID="97643b11a6f359931f02456a01b91c52d629d813a577720162aac03918d00eac" exitCode=0 Sep 09 14:33:48.160857 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.160525 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" event={"ID":"af356e02-73e3-4111-a82b-e1462c92982b","Type":"ContainerDied","Data":"97643b11a6f359931f02456a01b91c52d629d813a577720162aac03918d00eac"} Sep 09 14:33:48.907888 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.907860 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k2jgj"] Sep 09 14:33:48.908032 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.907980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:48.908109 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:48.908082 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:48.908679 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.908657 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5sm4r"] Sep 09 14:33:48.908782 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.908748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:48.908867 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:48.908842 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:48.909183 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.909160 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fs9bf"] Sep 09 14:33:48.909276 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:48.909265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:48.909405 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:48.909383 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:49.976753 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:49.976720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:49.977418 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:49.976818 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:49.977418 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:49.976877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:49.977418 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:49.976965 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:50.977898 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:50.977742 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:50.978270 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:50.977962 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:51.170415 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:51.170306 2578 generic.go:356] "Generic (PLEG): container finished" podID="af356e02-73e3-4111-a82b-e1462c92982b" containerID="ba7ca41e91abcd96a43a2f63f2b92f8f24da88e59dd51f2000af8d6e55ab10c9" exitCode=0 Sep 09 14:33:51.170530 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:51.170397 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" event={"ID":"af356e02-73e3-4111-a82b-e1462c92982b","Type":"ContainerDied","Data":"ba7ca41e91abcd96a43a2f63f2b92f8f24da88e59dd51f2000af8d6e55ab10c9"} Sep 09 14:33:51.976496 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:51.976463 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:51.976669 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:51.976463 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:51.976669 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:51.976558 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:51.976769 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:51.976666 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:52.976617 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:52.976592 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:52.977051 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:52.976699 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:33:53.177024 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:53.176983 2578 generic.go:356] "Generic (PLEG): container finished" podID="af356e02-73e3-4111-a82b-e1462c92982b" containerID="a2766596f2c8845b7130e34b8a52b2eec04e363f2546966484f70b12014966b8" exitCode=0 Sep 09 14:33:53.177181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:53.177051 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" event={"ID":"af356e02-73e3-4111-a82b-e1462c92982b","Type":"ContainerDied","Data":"a2766596f2c8845b7130e34b8a52b2eec04e363f2546966484f70b12014966b8"} Sep 09 14:33:53.976822 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:53.976788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:53.977281 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:53.976905 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k2jgj" podUID="ddac112e-73f0-4043-a739-7dd8b01527aa" Sep 09 14:33:53.977281 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:53.976964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:53.977281 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:53.977069 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sm4r" podUID="e2f50ed1-8a0b-4bff-a86c-c46232240402" Sep 09 14:33:54.486820 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.486781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:54.487032 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.486959 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 14:33:54.487032 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.486982 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 14:33:54.487032 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.486996 2578 projected.go:196] Error preparing data for projected volume kube-api-access-rnbbx for pod openshift-network-diagnostics/network-check-target-5sm4r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:54.487200 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.487056 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx podName:e2f50ed1-8a0b-4bff-a86c-c46232240402 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:26.487039358 +0000 UTC m=+66.153845402 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rnbbx" (UniqueName: "kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx") pod "network-check-target-5sm4r" (UID: "e2f50ed1-8a0b-4bff-a86c-c46232240402") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 14:33:54.560049 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.560020 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-173.ec2.internal" event="NodeReady" Sep 09 14:33:54.560249 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.560177 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 14:33:54.587246 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.587164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:54.587246 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.587203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:54.587474 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.587294 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:54.587474 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.587301 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:54.587474 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.587372 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:26.587336107 +0000 UTC m=+66.254142132 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 14:33:54.587474 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.587391 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret podName:ddac112e-73f0-4043-a739-7dd8b01527aa nodeName:}" failed. No retries permitted until 2026-09-09 14:34:26.587382268 +0000 UTC m=+66.254188297 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret") pod "global-pull-secret-syncer-k2jgj" (UID: "ddac112e-73f0-4043-a739-7dd8b01527aa") : object "kube-system"/"original-pull-secret" not registered Sep 09 14:33:54.603877 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.603851 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9v8sj"] Sep 09 14:33:54.623062 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.623040 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5ntb8"] Sep 09 14:33:54.623199 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.623180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.626749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.626726 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btvxq\"" Sep 09 14:33:54.627120 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.627083 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 14:33:54.627359 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.627332 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 14:33:54.643006 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.642976 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9v8sj"] Sep 09 14:33:54.643006 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.643008 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5ntb8"] Sep 09 14:33:54.643126 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.643096 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5ntb8" Sep 09 14:33:54.645692 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.645633 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 14:33:54.645692 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.645680 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dtd7b\"" Sep 09 14:33:54.645860 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.645733 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 14:33:54.645860 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.645683 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 14:33:54.789174 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.789146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mjh\" (UniqueName: \"kubernetes.io/projected/e2994b71-2a39-42b9-bd8c-1b55c03259b0-kube-api-access-66mjh\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.789428 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.789184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ef9739c-da6d-450b-be72-a2d7c60b9027-cert\") pod \"ingress-canary-5ntb8\" (UID: \"9ef9739c-da6d-450b-be72-a2d7c60b9027\") " pod="openshift-ingress-canary/ingress-canary-5ntb8" Sep 09 14:33:54.789428 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.789208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.789428 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.789297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgntt\" (UniqueName: \"kubernetes.io/projected/9ef9739c-da6d-450b-be72-a2d7c60b9027-kube-api-access-kgntt\") pod \"ingress-canary-5ntb8\" (UID: \"9ef9739c-da6d-450b-be72-a2d7c60b9027\") " pod="openshift-ingress-canary/ingress-canary-5ntb8" Sep 09 14:33:54.789428 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.789335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2994b71-2a39-42b9-bd8c-1b55c03259b0-tmp-dir\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.789428 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.789401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2994b71-2a39-42b9-bd8c-1b55c03259b0-config-volume\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.890219 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.890132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ef9739c-da6d-450b-be72-a2d7c60b9027-cert\") pod \"ingress-canary-5ntb8\" (UID: \"9ef9739c-da6d-450b-be72-a2d7c60b9027\") " pod="openshift-ingress-canary/ingress-canary-5ntb8" Sep 09 14:33:54.890219 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.890189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.890500 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.890301 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:54.890500 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:54.890385 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls podName:e2994b71-2a39-42b9-bd8c-1b55c03259b0 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:55.390366753 +0000 UTC m=+35.057172794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls") pod "dns-default-9v8sj" (UID: "e2994b71-2a39-42b9-bd8c-1b55c03259b0") : secret "dns-default-metrics-tls" not found Sep 09 14:33:54.890500 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.890300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgntt\" (UniqueName: \"kubernetes.io/projected/9ef9739c-da6d-450b-be72-a2d7c60b9027-kube-api-access-kgntt\") pod \"ingress-canary-5ntb8\" (UID: \"9ef9739c-da6d-450b-be72-a2d7c60b9027\") " pod="openshift-ingress-canary/ingress-canary-5ntb8" Sep 09 14:33:54.890500 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.890472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2994b71-2a39-42b9-bd8c-1b55c03259b0-tmp-dir\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.890685 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.890524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2994b71-2a39-42b9-bd8c-1b55c03259b0-config-volume\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.890685 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.890581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mjh\" (UniqueName: \"kubernetes.io/projected/e2994b71-2a39-42b9-bd8c-1b55c03259b0-kube-api-access-66mjh\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.890929 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.890900 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2994b71-2a39-42b9-bd8c-1b55c03259b0-tmp-dir\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.891125 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.891104 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2994b71-2a39-42b9-bd8c-1b55c03259b0-config-volume\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.895041 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.895014 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ef9739c-da6d-450b-be72-a2d7c60b9027-cert\") pod \"ingress-canary-5ntb8\" (UID: \"9ef9739c-da6d-450b-be72-a2d7c60b9027\") " pod="openshift-ingress-canary/ingress-canary-5ntb8" Sep 09 14:33:54.901317 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.901282 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgntt\" (UniqueName: \"kubernetes.io/projected/9ef9739c-da6d-450b-be72-a2d7c60b9027-kube-api-access-kgntt\") pod \"ingress-canary-5ntb8\" (UID: \"9ef9739c-da6d-450b-be72-a2d7c60b9027\") " pod="openshift-ingress-canary/ingress-canary-5ntb8" Sep 09 14:33:54.901622 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.901607 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mjh\" (UniqueName: \"kubernetes.io/projected/e2994b71-2a39-42b9-bd8c-1b55c03259b0-kube-api-access-66mjh\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:54.952847 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.952815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5ntb8" Sep 09 14:33:54.980080 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.980060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:33:54.982906 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.982869 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 14:33:54.983024 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:54.982935 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bjh9b\"" Sep 09 14:33:55.269770 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:55.269580 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5ntb8"] Sep 09 14:33:55.273752 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:33:55.273701 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef9739c_da6d_450b_be72_a2d7c60b9027.slice/crio-fd70b865369cf9bf5a174fb340023416c1819de9768f5412e7e3fbddd044d892 WatchSource:0}: Error finding container fd70b865369cf9bf5a174fb340023416c1819de9768f5412e7e3fbddd044d892: Status 404 returned error can't find the container with id fd70b865369cf9bf5a174fb340023416c1819de9768f5412e7e3fbddd044d892 Sep 09 14:33:55.394819 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:55.394765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:55.395022 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:55.394947 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:55.395090 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:55.395036 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls podName:e2994b71-2a39-42b9-bd8c-1b55c03259b0 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:56.395015173 +0000 UTC m=+36.061821199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls") pod "dns-default-9v8sj" (UID: "e2994b71-2a39-42b9-bd8c-1b55c03259b0") : secret "dns-default-metrics-tls" not found Sep 09 14:33:55.976544 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:55.976504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:33:55.976544 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:55.976531 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:33:55.979158 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:55.979118 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 14:33:55.979837 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:55.979816 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 14:33:55.979837 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:55.979831 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l56jk\"" Sep 09 14:33:55.980144 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:55.980122 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 14:33:56.186292 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:56.186252 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5ntb8" event={"ID":"9ef9739c-da6d-450b-be72-a2d7c60b9027","Type":"ContainerStarted","Data":"fd70b865369cf9bf5a174fb340023416c1819de9768f5412e7e3fbddd044d892"} Sep 09 14:33:56.403510 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:56.403422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:56.403675 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:56.403595 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:56.403739 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:56.403675 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls podName:e2994b71-2a39-42b9-bd8c-1b55c03259b0 nodeName:}" failed. No retries permitted until 2026-09-09 14:33:58.403653507 +0000 UTC m=+38.070459537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls") pod "dns-default-9v8sj" (UID: "e2994b71-2a39-42b9-bd8c-1b55c03259b0") : secret "dns-default-metrics-tls" not found Sep 09 14:33:58.419958 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:58.419934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:33:58.420480 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:58.420102 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:33:58.420480 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:33:58.420170 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls podName:e2994b71-2a39-42b9-bd8c-1b55c03259b0 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:02.420151426 +0000 UTC m=+42.086957465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls") pod "dns-default-9v8sj" (UID: "e2994b71-2a39-42b9-bd8c-1b55c03259b0") : secret "dns-default-metrics-tls" not found Sep 09 14:33:59.194297 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:59.194256 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5ntb8" event={"ID":"9ef9739c-da6d-450b-be72-a2d7c60b9027","Type":"ContainerStarted","Data":"20e6a3680d06eaa4406cda3943410236dd5a6f4d2db7f7c3d5a877a6cd18f60e"} Sep 09 14:33:59.196953 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:59.196920 2578 generic.go:356] "Generic (PLEG): container finished" podID="af356e02-73e3-4111-a82b-e1462c92982b" containerID="578e58fba256b2c3c0d56af396037a065796c3d5069865d11e4c19691271495e" exitCode=0 Sep 09 14:33:59.197072 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:59.196961 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" event={"ID":"af356e02-73e3-4111-a82b-e1462c92982b","Type":"ContainerDied","Data":"578e58fba256b2c3c0d56af396037a065796c3d5069865d11e4c19691271495e"} Sep 09 14:33:59.228000 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:33:59.227951 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5ntb8" podStartSLOduration=2.095004539 podStartE2EDuration="5.227934552s" totalImagesPullingTime="3.132930013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:33:59.209262854 +0000 UTC m=+38.876068901" watchObservedRunningTime="2026-09-09 14:33:59.227934552 +0000 UTC m=+38.894740597" Sep 09 14:34:00.202879 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:00.202843 2578 generic.go:356] "Generic (PLEG): container finished" podID="af356e02-73e3-4111-a82b-e1462c92982b" containerID="335ca383ec4a1d7e7f6cd7245e2967964c035557a836f238156045f0239c589d" exitCode=0 Sep 09 14:34:00.203480 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:00.202936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" event={"ID":"af356e02-73e3-4111-a82b-e1462c92982b","Type":"ContainerDied","Data":"335ca383ec4a1d7e7f6cd7245e2967964c035557a836f238156045f0239c589d"} Sep 09 14:34:01.207858 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:01.207814 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" event={"ID":"af356e02-73e3-4111-a82b-e1462c92982b","Type":"ContainerStarted","Data":"4899a1d3c3b7a0c87df4323c10dbdb179c892df66155f63052e212790e006b52"} Sep 09 14:34:01.234280 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:01.234233 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rl6h9" podStartSLOduration=8.089555493 podStartE2EDuration="40.234218847s" totalImagesPullingTime="32.144663354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:34:01.23321386 +0000 UTC m=+40.900019916" watchObservedRunningTime="2026-09-09 14:34:01.234218847 +0000 UTC m=+40.901024893" Sep 09 14:34:02.448310 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:02.448273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:34:02.448690 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:02.448425 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:34:02.448690 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:02.448489 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls podName:e2994b71-2a39-42b9-bd8c-1b55c03259b0 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:10.448473333 +0000 UTC m=+50.115279357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls") pod "dns-default-9v8sj" (UID: "e2994b71-2a39-42b9-bd8c-1b55c03259b0") : secret "dns-default-metrics-tls" not found Sep 09 14:34:10.504006 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:10.503965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:34:10.504343 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:10.504136 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:34:10.504343 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:10.504214 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls podName:e2994b71-2a39-42b9-bd8c-1b55c03259b0 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:26.504199008 +0000 UTC m=+66.171005032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls") pod "dns-default-9v8sj" (UID: "e2994b71-2a39-42b9-bd8c-1b55c03259b0") : secret "dns-default-metrics-tls" not found Sep 09 14:34:19.172861 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:19.172832 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66bb4" Sep 09 14:34:26.504321 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.504274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:34:26.504321 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.504333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:34:26.504780 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:26.504449 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:34:26.504780 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:26.504499 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls podName:e2994b71-2a39-42b9-bd8c-1b55c03259b0 nodeName:}" failed. No retries permitted until 2026-09-09 14:34:58.504484837 +0000 UTC m=+98.171290861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls") pod "dns-default-9v8sj" (UID: "e2994b71-2a39-42b9-bd8c-1b55c03259b0") : secret "dns-default-metrics-tls" not found Sep 09 14:34:26.507078 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.507060 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 14:34:26.516684 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.516659 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 14:34:26.527648 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.527622 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnbbx\" (UniqueName: \"kubernetes.io/projected/e2f50ed1-8a0b-4bff-a86c-c46232240402-kube-api-access-rnbbx\") pod \"network-check-target-5sm4r\" (UID: \"e2f50ed1-8a0b-4bff-a86c-c46232240402\") " pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:34:26.592593 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.592570 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l56jk\"" Sep 09 14:34:26.600796 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.600778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:34:26.605136 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.605111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:34:26.605224 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.605143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:34:26.607474 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.607459 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 14:34:26.607631 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.607616 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 14:34:26.615589 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:26.615562 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 14:34:26.615678 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:26.615629 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:35:30.615609805 +0000 UTC m=+130.282415842 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : secret "metrics-daemon-secret" not found Sep 09 14:34:26.617754 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.617733 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ddac112e-73f0-4043-a739-7dd8b01527aa-original-pull-secret\") pod \"global-pull-secret-syncer-k2jgj\" (UID: \"ddac112e-73f0-4043-a739-7dd8b01527aa\") " pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:34:26.871418 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.871321 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5sm4r"] Sep 09 14:34:26.874760 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:34:26.874735 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f50ed1_8a0b_4bff_a86c_c46232240402.slice/crio-4f495e4ec6483485084b44a0860cd994d84f629d1471a2df17d621f5b8f0613d WatchSource:0}: Error finding container 4f495e4ec6483485084b44a0860cd994d84f629d1471a2df17d621f5b8f0613d: Status 404 returned error can't find the container with id 4f495e4ec6483485084b44a0860cd994d84f629d1471a2df17d621f5b8f0613d Sep 09 14:34:26.896435 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:26.896417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k2jgj" Sep 09 14:34:27.162775 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:27.162700 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k2jgj"] Sep 09 14:34:27.166071 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:34:27.166045 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddac112e_73f0_4043_a739_7dd8b01527aa.slice/crio-115222acbca21af4f237f192ae35c21b37af324026f143764c0b866eadfc97e3 WatchSource:0}: Error finding container 115222acbca21af4f237f192ae35c21b37af324026f143764c0b866eadfc97e3: Status 404 returned error can't find the container with id 115222acbca21af4f237f192ae35c21b37af324026f143764c0b866eadfc97e3 Sep 09 14:34:27.270233 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:27.270202 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k2jgj" event={"ID":"ddac112e-73f0-4043-a739-7dd8b01527aa","Type":"ContainerStarted","Data":"115222acbca21af4f237f192ae35c21b37af324026f143764c0b866eadfc97e3"} Sep 09 14:34:27.271293 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:27.271265 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5sm4r" event={"ID":"e2f50ed1-8a0b-4bff-a86c-c46232240402","Type":"ContainerStarted","Data":"4f495e4ec6483485084b44a0860cd994d84f629d1471a2df17d621f5b8f0613d"} Sep 09 14:34:32.285527 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:32.285491 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k2jgj" event={"ID":"ddac112e-73f0-4043-a739-7dd8b01527aa","Type":"ContainerStarted","Data":"fd89d024266db0c20ef1a4c084f1f1f51f9472d42d8b1055d0dc639fe668efae"} Sep 09 14:34:32.287341 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:32.287319 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5sm4r" event={"ID":"e2f50ed1-8a0b-4bff-a86c-c46232240402","Type":"ContainerStarted","Data":"e3f48a651c904dda3684244130597c58739abdeec4cb275a9ea8d59e6538e1e6"} Sep 09 14:34:32.287505 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:32.287410 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:34:32.318404 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:32.318329 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k2jgj" podStartSLOduration=66.206852606 podStartE2EDuration="1m10.318315629s" totalImagesPullingTime="4.111463023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:34:32.317864625 +0000 UTC m=+71.984670670" watchObservedRunningTime="2026-09-09 14:34:32.318315629 +0000 UTC m=+71.985121669" Sep 09 14:34:32.346653 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:32.346610 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5sm4r" podStartSLOduration=66.946483249 podStartE2EDuration="1m11.346596062s" totalImagesPullingTime="4.400112813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:34:32.34595227 +0000 UTC m=+72.012758316" watchObservedRunningTime="2026-09-09 14:34:32.346596062 +0000 UTC m=+72.013402107" Sep 09 14:34:36.199988 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.199948 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z"] Sep 09 14:34:36.202487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.202465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" Sep 09 14:34:36.206263 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.206238 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 09 14:34:36.206887 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.206864 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bpww9\"" Sep 09 14:34:36.206887 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.206876 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 14:34:36.207044 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.206949 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 14:34:36.207044 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.206959 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 14:34:36.209404 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.209383 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z"] Sep 09 14:34:36.269204 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.269178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6mr5\" (UniqueName: \"kubernetes.io/projected/e0c3af09-b24c-4b32-9e8b-bac5f2da1f60-kube-api-access-q6mr5\") pod \"managed-serviceaccount-addon-agent-558df45b68-zk68z\" (UID: \"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" Sep 09 14:34:36.269305 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.269236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0c3af09-b24c-4b32-9e8b-bac5f2da1f60-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-558df45b68-zk68z\" (UID: \"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" Sep 09 14:34:36.370304 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.370283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6mr5\" (UniqueName: \"kubernetes.io/projected/e0c3af09-b24c-4b32-9e8b-bac5f2da1f60-kube-api-access-q6mr5\") pod \"managed-serviceaccount-addon-agent-558df45b68-zk68z\" (UID: \"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" Sep 09 14:34:36.370424 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.370326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0c3af09-b24c-4b32-9e8b-bac5f2da1f60-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-558df45b68-zk68z\" (UID: \"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" Sep 09 14:34:36.372708 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.372689 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0c3af09-b24c-4b32-9e8b-bac5f2da1f60-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-558df45b68-zk68z\" (UID: \"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" Sep 09 14:34:36.378152 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.378132 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6mr5\" (UniqueName: \"kubernetes.io/projected/e0c3af09-b24c-4b32-9e8b-bac5f2da1f60-kube-api-access-q6mr5\") pod \"managed-serviceaccount-addon-agent-558df45b68-zk68z\" (UID: \"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" Sep 09 14:34:36.527406 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.527376 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" Sep 09 14:34:36.795992 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:36.795920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z"] Sep 09 14:34:36.799945 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:34:36.799919 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0c3af09_b24c_4b32_9e8b_bac5f2da1f60.slice/crio-c415dc1075141d0a42af799f3f2551055c69a597d762e8ac14fc7f10ea6c8901 WatchSource:0}: Error finding container c415dc1075141d0a42af799f3f2551055c69a597d762e8ac14fc7f10ea6c8901: Status 404 returned error can't find the container with id c415dc1075141d0a42af799f3f2551055c69a597d762e8ac14fc7f10ea6c8901 Sep 09 14:34:37.300505 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:37.300468 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" event={"ID":"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60","Type":"ContainerStarted","Data":"c415dc1075141d0a42af799f3f2551055c69a597d762e8ac14fc7f10ea6c8901"} Sep 09 14:34:41.311248 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:41.311214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" event={"ID":"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60","Type":"ContainerStarted","Data":"f6ad1e06797b37bd9ab29facc6e57cdb123d9dd25fadcfc095e88813700535ee"} Sep 09 14:34:58.522330 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:34:58.522298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:34:58.522740 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:58.522447 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:34:58.522740 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:34:58.522523 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls podName:e2994b71-2a39-42b9-bd8c-1b55c03259b0 nodeName:}" failed. No retries permitted until 2026-09-09 14:36:02.522505103 +0000 UTC m=+162.189311127 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls") pod "dns-default-9v8sj" (UID: "e2994b71-2a39-42b9-bd8c-1b55c03259b0") : secret "dns-default-metrics-tls" not found Sep 09 14:35:03.292942 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:35:03.292910 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5sm4r" Sep 09 14:35:03.309866 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:35:03.309819 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" podStartSLOduration=23.367979083 podStartE2EDuration="27.309804449s" totalImagesPullingTime="3.941825366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:34:41.330436445 +0000 UTC m=+80.997242492" watchObservedRunningTime="2026-09-09 14:35:03.309804449 +0000 UTC m=+102.976610494" Sep 09 14:35:30.638940 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:35:30.638889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:35:30.639444 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:35:30.639040 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 14:35:30.639444 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:35:30.639107 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:32.639091506 +0000 UTC m=+252.305897533 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : secret "metrics-daemon-secret" not found Sep 09 14:35:57.634160 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:35:57.634121 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9v8sj" podUID="e2994b71-2a39-42b9-bd8c-1b55c03259b0" Sep 09 14:35:58.004487 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:35:58.004444 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:35:58.495220 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:35:58.495192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9v8sj" Sep 09 14:36:02.558157 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:36:02.558121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:36:02.558580 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:36:02.558263 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 14:36:02.558580 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:36:02.558337 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls podName:e2994b71-2a39-42b9-bd8c-1b55c03259b0 nodeName:}" failed. No retries permitted until 2026-09-09 14:38:04.558319844 +0000 UTC m=+284.225125867 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls") pod "dns-default-9v8sj" (UID: "e2994b71-2a39-42b9-bd8c-1b55c03259b0") : secret "dns-default-metrics-tls" not found Sep 09 14:36:11.977025 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:36:11.976990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:36:41.599770 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:36:41.599736 2578 generic.go:356] "Generic (PLEG): container finished" podID="e0c3af09-b24c-4b32-9e8b-bac5f2da1f60" containerID="f6ad1e06797b37bd9ab29facc6e57cdb123d9dd25fadcfc095e88813700535ee" exitCode=255 Sep 09 14:36:41.600166 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:36:41.599817 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" event={"ID":"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60","Type":"ContainerDied","Data":"f6ad1e06797b37bd9ab29facc6e57cdb123d9dd25fadcfc095e88813700535ee"} Sep 09 14:36:41.605377 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:36:41.605359 2578 scope.go:117] "RemoveContainer" containerID="f6ad1e06797b37bd9ab29facc6e57cdb123d9dd25fadcfc095e88813700535ee" Sep 09 14:36:42.604290 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:36:42.604256 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-558df45b68-zk68z" event={"ID":"e0c3af09-b24c-4b32-9e8b-bac5f2da1f60","Type":"ContainerStarted","Data":"84c6b3a7d0a97a3e1862cd462ce577a1366369f83a86e7b05ebcfcde950c901b"} Sep 09 14:37:31.567577 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.567539 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-574b7bc7d5-5hcss"] Sep 09 14:37:31.570712 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.570691 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.573714 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.573690 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 14:37:31.573819 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.573744 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 14:37:31.573819 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.573762 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m98bc\"" Sep 09 14:37:31.575036 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.575021 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 14:37:31.583388 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.583369 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 14:37:31.592039 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.592020 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv"] Sep 09 14:37:31.594750 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.594733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:31.598096 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.598077 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 14:37:31.598096 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.598092 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 14:37:31.598335 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.598319 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 09 14:37:31.598498 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.598482 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-n7g7g"] Sep 09 14:37:31.599373 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.599335 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 09 14:37:31.599453 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.599335 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-56prc\"" Sep 09 14:37:31.601185 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.601167 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-594c74b48d-2brjb"] Sep 09 14:37:31.601283 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.601269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.603798 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.603781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 09 14:37:31.603872 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.603851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.604112 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.604093 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fvm54\"" Sep 09 14:37:31.604308 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.604292 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 09 14:37:31.604416 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.604293 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 09 14:37:31.604794 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.604776 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 09 14:37:31.606799 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.606781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 09 14:37:31.611041 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.611022 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 09 14:37:31.611843 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.611822 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-574b7bc7d5-5hcss"] Sep 09 14:37:31.620207 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.620184 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv"] Sep 09 14:37:31.629819 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.629797 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 09 14:37:31.629910 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.629820 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 09 14:37:31.629964 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.629918 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gb885\"" Sep 09 14:37:31.630045 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.630030 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 09 14:37:31.630090 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.630064 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 14:37:31.634619 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.634595 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 14:37:31.634919 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.634902 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-n7g7g"] Sep 09 14:37:31.637941 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.637911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-bound-sa-token\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.638025 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.637955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-certificates\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.638025 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.637981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-installation-pull-secrets\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.638025 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.638005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vsh\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-kube-api-access-55vsh\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.638134 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.638085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-trusted-ca\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.638173 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.638139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-image-registry-private-configuration\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.638208 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.638177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.638208 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.638199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ee8a629-5cfe-4f0a-836b-725b2a469dee-ca-trust-extracted\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.642001 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.641979 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-594c74b48d-2brjb"] Sep 09 14:37:31.666750 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.666724 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv"] Sep 09 14:37:31.670490 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.670464 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289"] Sep 09 14:37:31.670583 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.670574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:31.673112 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.673094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289" Sep 09 14:37:31.674862 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.674846 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 09 14:37:31.675223 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.675208 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 09 14:37:31.675333 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.675319 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 09 14:37:31.676027 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.676012 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 09 14:37:31.676317 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.676304 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 09 14:37:31.676707 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.676694 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hjtbm\"" Sep 09 14:37:31.677197 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.677172 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q9jvp\"" Sep 09 14:37:31.696815 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.696794 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289"] Sep 09 14:37:31.697427 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.697408 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv"] Sep 09 14:37:31.738936 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.738915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.739028 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.738941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9zs\" (UniqueName: \"kubernetes.io/projected/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-kube-api-access-9b9zs\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.739028 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.738959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnd85\" (UniqueName: \"kubernetes.io/projected/e0ce45bb-2ee5-4e19-92ef-134ca514e125-kube-api-access-nnd85\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:31.739028 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.738975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-serving-cert\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.739028 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.738999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-image-registry-private-configuration\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.739154 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52d3841d-c086-4a5a-99dd-6f98adffbfd9-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:31.739154 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.739215 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ee8a629-5cfe-4f0a-836b-725b2a469dee-ca-trust-extracted\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.739215 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.739190 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:37:31.739215 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.739202 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b7bc7d5-5hcss: secret "image-registry-tls" not found Sep 09 14:37:31.739215 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59dr\" (UniqueName: \"kubernetes.io/projected/52d3841d-c086-4a5a-99dd-6f98adffbfd9-kube-api-access-n59dr\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:31.739379 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-bound-sa-token\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.739379 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.739379 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.739297 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls podName:2ee8a629-5cfe-4f0a-836b-725b2a469dee nodeName:}" failed. No retries permitted until 2026-09-09 14:37:32.239274042 +0000 UTC m=+251.906080072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls") pod "image-registry-574b7bc7d5-5hcss" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee") : secret "image-registry-tls" not found Sep 09 14:37:31.739379 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-certificates\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.739582 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vsh\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-kube-api-access-55vsh\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.739582 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:31.739582 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-trusted-ca\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.739582 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sm7s\" (UniqueName: \"kubernetes.io/projected/f739c355-396f-4032-86b7-3ad0ccc92c48-kube-api-access-4sm7s\") pod \"volume-data-source-validator-695c895bd6-gj289\" (UID: \"f739c355-396f-4032-86b7-3ad0ccc92c48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289" Sep 09 14:37:31.739582 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:31.739582 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-trusted-ca\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.739885 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739606 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ee8a629-5cfe-4f0a-836b-725b2a469dee-ca-trust-extracted\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.739885 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s22wc\" (UniqueName: \"kubernetes.io/projected/712c3b38-ef28-4237-8de9-1d9a6058866c-kube-api-access-s22wc\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.739885 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-stats-auth\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.739885 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-installation-pull-secrets\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.739885 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-default-certificate\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.739885 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739802 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-config\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.740097 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.739928 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-certificates\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.740478 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.740455 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-trusted-ca\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.741758 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.741738 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-image-registry-private-configuration\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.741967 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.741948 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-installation-pull-secrets\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.759108 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.759083 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h"] Sep 09 14:37:31.762111 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.762093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.765569 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.765544 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 09 14:37:31.765569 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.765563 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 09 14:37:31.765999 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.765982 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qwdpk\"" Sep 09 14:37:31.766069 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.766012 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 09 14:37:31.767432 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.767412 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vsh\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-kube-api-access-55vsh\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.769138 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.769117 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 09 14:37:31.771069 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.771051 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-bound-sa-token\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:31.788868 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.788849 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h"] Sep 09 14:37:31.841151 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841080 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59dr\" (UniqueName: \"kubernetes.io/projected/52d3841d-c086-4a5a-99dd-6f98adffbfd9-kube-api-access-n59dr\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:31.841151 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c295v\" (UniqueName: \"kubernetes.io/projected/ba32ff86-2186-4f15-84f4-8b2c63867723-kube-api-access-c295v\") pod \"service-ca-operator-859f94c4b-2mw8h\" (UID: \"ba32ff86-2186-4f15-84f4-8b2c63867723\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.841342 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.841342 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:31.841342 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sm7s\" (UniqueName: \"kubernetes.io/projected/f739c355-396f-4032-86b7-3ad0ccc92c48-kube-api-access-4sm7s\") pod \"volume-data-source-validator-695c895bd6-gj289\" (UID: \"f739c355-396f-4032-86b7-3ad0ccc92c48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289" Sep 09 14:37:31.841342 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.841299 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:31.841342 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:31.841342 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.841329 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:32.341311768 +0000 UTC m=+252.008117807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : configmap references non-existent config key: service-ca.crt Sep 09 14:37:31.841690 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.841379 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls podName:52d3841d-c086-4a5a-99dd-6f98adffbfd9 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:32.341365277 +0000 UTC m=+252.008171306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9tkzv" (UID: "52d3841d-c086-4a5a-99dd-6f98adffbfd9") : secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:31.841690 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.841396 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 14:37:31.841690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-trusted-ca\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.841690 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.841424 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls podName:e0ce45bb-2ee5-4e19-92ef-134ca514e125 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:32.341412884 +0000 UTC m=+252.008218914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-qfvdv" (UID: "e0ce45bb-2ee5-4e19-92ef-134ca514e125") : secret "samples-operator-tls" not found Sep 09 14:37:31.841690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba32ff86-2186-4f15-84f4-8b2c63867723-config\") pod \"service-ca-operator-859f94c4b-2mw8h\" (UID: \"ba32ff86-2186-4f15-84f4-8b2c63867723\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.841690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba32ff86-2186-4f15-84f4-8b2c63867723-serving-cert\") pod \"service-ca-operator-859f94c4b-2mw8h\" (UID: \"ba32ff86-2186-4f15-84f4-8b2c63867723\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.841690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s22wc\" (UniqueName: \"kubernetes.io/projected/712c3b38-ef28-4237-8de9-1d9a6058866c-kube-api-access-s22wc\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.841690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-stats-auth\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.841690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-default-certificate\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.842129 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-config\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.842129 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.842129 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9zs\" (UniqueName: \"kubernetes.io/projected/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-kube-api-access-9b9zs\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.842129 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd85\" (UniqueName: \"kubernetes.io/projected/e0ce45bb-2ee5-4e19-92ef-134ca514e125-kube-api-access-nnd85\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:31.842129 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.841844 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 14:37:31.842129 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:31.841898 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:32.34188743 +0000 UTC m=+252.008693458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : secret "router-metrics-certs-default" not found Sep 09 14:37:31.842129 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.841957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-serving-cert\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.842129 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.842027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52d3841d-c086-4a5a-99dd-6f98adffbfd9-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:31.842522 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.842258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-trusted-ca\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.842522 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.842370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-config\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.842734 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.842714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52d3841d-c086-4a5a-99dd-6f98adffbfd9-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:31.844086 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.844070 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-serving-cert\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.844173 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.844135 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-stats-auth\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.844215 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.844191 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-default-certificate\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.857543 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.857519 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59dr\" (UniqueName: \"kubernetes.io/projected/52d3841d-c086-4a5a-99dd-6f98adffbfd9-kube-api-access-n59dr\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:31.860185 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.860163 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s22wc\" (UniqueName: \"kubernetes.io/projected/712c3b38-ef28-4237-8de9-1d9a6058866c-kube-api-access-s22wc\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:31.860483 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.860464 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sm7s\" (UniqueName: \"kubernetes.io/projected/f739c355-396f-4032-86b7-3ad0ccc92c48-kube-api-access-4sm7s\") pod \"volume-data-source-validator-695c895bd6-gj289\" (UID: \"f739c355-396f-4032-86b7-3ad0ccc92c48\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289" Sep 09 14:37:31.860663 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.860646 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd85\" (UniqueName: \"kubernetes.io/projected/e0ce45bb-2ee5-4e19-92ef-134ca514e125-kube-api-access-nnd85\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:31.860863 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.860847 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9zs\" (UniqueName: \"kubernetes.io/projected/96887fc2-e8bc-4ae0-b45d-96f012fbcf07-kube-api-access-9b9zs\") pod \"console-operator-85b454874b-n7g7g\" (UID: \"96887fc2-e8bc-4ae0-b45d-96f012fbcf07\") " pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.912415 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.912381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:31.942833 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.942809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c295v\" (UniqueName: \"kubernetes.io/projected/ba32ff86-2186-4f15-84f4-8b2c63867723-kube-api-access-c295v\") pod \"service-ca-operator-859f94c4b-2mw8h\" (UID: \"ba32ff86-2186-4f15-84f4-8b2c63867723\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.942941 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.942895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba32ff86-2186-4f15-84f4-8b2c63867723-config\") pod \"service-ca-operator-859f94c4b-2mw8h\" (UID: \"ba32ff86-2186-4f15-84f4-8b2c63867723\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.942941 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.942918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba32ff86-2186-4f15-84f4-8b2c63867723-serving-cert\") pod \"service-ca-operator-859f94c4b-2mw8h\" (UID: \"ba32ff86-2186-4f15-84f4-8b2c63867723\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.943428 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.943405 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba32ff86-2186-4f15-84f4-8b2c63867723-config\") pod \"service-ca-operator-859f94c4b-2mw8h\" (UID: \"ba32ff86-2186-4f15-84f4-8b2c63867723\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.944981 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.944951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba32ff86-2186-4f15-84f4-8b2c63867723-serving-cert\") pod \"service-ca-operator-859f94c4b-2mw8h\" (UID: \"ba32ff86-2186-4f15-84f4-8b2c63867723\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.953364 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.953330 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c295v\" (UniqueName: \"kubernetes.io/projected/ba32ff86-2186-4f15-84f4-8b2c63867723-kube-api-access-c295v\") pod \"service-ca-operator-859f94c4b-2mw8h\" (UID: \"ba32ff86-2186-4f15-84f4-8b2c63867723\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:31.984819 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:31.984792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289" Sep 09 14:37:32.078935 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.078909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" Sep 09 14:37:32.183947 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.183920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-n7g7g"] Sep 09 14:37:32.186531 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:37:32.186505 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96887fc2_e8bc_4ae0_b45d_96f012fbcf07.slice/crio-1a5f9fabeedad76208f9662935763d9e336b3453400d85fda8d35742ae8673da WatchSource:0}: Error finding container 1a5f9fabeedad76208f9662935763d9e336b3453400d85fda8d35742ae8673da: Status 404 returned error can't find the container with id 1a5f9fabeedad76208f9662935763d9e336b3453400d85fda8d35742ae8673da Sep 09 14:37:32.245760 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.245738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:32.245862 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.245850 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:37:32.245910 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.245863 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b7bc7d5-5hcss: secret "image-registry-tls" not found Sep 09 14:37:32.245910 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.245904 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls podName:2ee8a629-5cfe-4f0a-836b-725b2a469dee nodeName:}" failed. No retries permitted until 2026-09-09 14:37:33.245893067 +0000 UTC m=+252.912699091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls") pod "image-registry-574b7bc7d5-5hcss" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee") : secret "image-registry-tls" not found Sep 09 14:37:32.253787 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.253767 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289"] Sep 09 14:37:32.255913 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:37:32.255891 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf739c355_396f_4032_86b7_3ad0ccc92c48.slice/crio-3472c25c9a291c313e54e8ad1b28ad2775fd02abaee521cff8d28e50a62c5448 WatchSource:0}: Error finding container 3472c25c9a291c313e54e8ad1b28ad2775fd02abaee521cff8d28e50a62c5448: Status 404 returned error can't find the container with id 3472c25c9a291c313e54e8ad1b28ad2775fd02abaee521cff8d28e50a62c5448 Sep 09 14:37:32.343447 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.343416 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h"] Sep 09 14:37:32.346148 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.346132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:32.346213 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.346197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:32.346262 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.346242 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 14:37:32.346308 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.346294 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls podName:e0ce45bb-2ee5-4e19-92ef-134ca514e125 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:33.346281115 +0000 UTC m=+253.013087144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-qfvdv" (UID: "e0ce45bb-2ee5-4e19-92ef-134ca514e125") : secret "samples-operator-tls" not found Sep 09 14:37:32.346389 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.346246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:32.346389 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.346322 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:33.346304743 +0000 UTC m=+253.013110787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : configmap references non-existent config key: service-ca.crt Sep 09 14:37:32.346389 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.346327 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 14:37:32.346389 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.346380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:32.346525 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.346397 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:33.346382577 +0000 UTC m=+253.013188617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : secret "router-metrics-certs-default" not found Sep 09 14:37:32.346525 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.346451 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:32.346525 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.346480 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls podName:52d3841d-c086-4a5a-99dd-6f98adffbfd9 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:33.346469295 +0000 UTC m=+253.013275319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9tkzv" (UID: "52d3841d-c086-4a5a-99dd-6f98adffbfd9") : secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:32.347068 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:37:32.347050 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba32ff86_2186_4f15_84f4_8b2c63867723.slice/crio-511cd40f61ddbeb99b4222b1667053b1e7ed6a4cc011ac3cd2dc048aea7f4463 WatchSource:0}: Error finding container 511cd40f61ddbeb99b4222b1667053b1e7ed6a4cc011ac3cd2dc048aea7f4463: Status 404 returned error can't find the container with id 511cd40f61ddbeb99b4222b1667053b1e7ed6a4cc011ac3cd2dc048aea7f4463 Sep 09 14:37:32.649120 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.649089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:37:32.649465 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.649235 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 14:37:32.649465 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:32.649333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs podName:c566cce4-7251-42bf-8696-f2505562d073 nodeName:}" failed. No retries permitted until 2026-09-09 14:39:34.649317258 +0000 UTC m=+374.316123283 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs") pod "network-metrics-daemon-fs9bf" (UID: "c566cce4-7251-42bf-8696-f2505562d073") : secret "metrics-daemon-secret" not found Sep 09 14:37:32.726585 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.726556 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289" event={"ID":"f739c355-396f-4032-86b7-3ad0ccc92c48","Type":"ContainerStarted","Data":"3472c25c9a291c313e54e8ad1b28ad2775fd02abaee521cff8d28e50a62c5448"} Sep 09 14:37:32.729532 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.729506 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" event={"ID":"96887fc2-e8bc-4ae0-b45d-96f012fbcf07","Type":"ContainerStarted","Data":"1a5f9fabeedad76208f9662935763d9e336b3453400d85fda8d35742ae8673da"} Sep 09 14:37:32.730510 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:32.730487 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" event={"ID":"ba32ff86-2186-4f15-84f4-8b2c63867723","Type":"ContainerStarted","Data":"511cd40f61ddbeb99b4222b1667053b1e7ed6a4cc011ac3cd2dc048aea7f4463"} Sep 09 14:37:33.253928 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:33.253865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:33.254108 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.254006 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:37:33.254108 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.254029 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b7bc7d5-5hcss: secret "image-registry-tls" not found Sep 09 14:37:33.254108 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.254098 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls podName:2ee8a629-5cfe-4f0a-836b-725b2a469dee nodeName:}" failed. No retries permitted until 2026-09-09 14:37:35.254077199 +0000 UTC m=+254.920883235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls") pod "image-registry-574b7bc7d5-5hcss" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee") : secret "image-registry-tls" not found Sep 09 14:37:33.355123 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:33.355090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:33.355268 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:33.355134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:33.355268 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:33.355193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:33.355268 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:33.355252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:33.355477 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.355279 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:33.355477 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.355324 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 14:37:33.355477 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.355334 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:35.355314081 +0000 UTC m=+255.022120108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : configmap references non-existent config key: service-ca.crt Sep 09 14:37:33.355477 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.355372 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls podName:52d3841d-c086-4a5a-99dd-6f98adffbfd9 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:35.355361403 +0000 UTC m=+255.022167433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9tkzv" (UID: "52d3841d-c086-4a5a-99dd-6f98adffbfd9") : secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:33.355477 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.355377 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 14:37:33.355477 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.355389 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls podName:e0ce45bb-2ee5-4e19-92ef-134ca514e125 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:35.355379315 +0000 UTC m=+255.022185343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-qfvdv" (UID: "e0ce45bb-2ee5-4e19-92ef-134ca514e125") : secret "samples-operator-tls" not found Sep 09 14:37:33.355477 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:33.355423 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:35.355406983 +0000 UTC m=+255.022213025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : secret "router-metrics-certs-default" not found Sep 09 14:37:34.740179 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:34.739846 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" event={"ID":"96887fc2-e8bc-4ae0-b45d-96f012fbcf07","Type":"ContainerStarted","Data":"56dd72d8a5a9ba071c852a58ed0463aac9352fd1f5dba48796a48ac8c2328777"} Sep 09 14:37:34.740179 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:34.740106 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:34.741549 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:34.741511 2578 patch_prober.go:28] interesting pod/console-operator-85b454874b-n7g7g container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/readyz\": dial tcp 10.134.0.11:8443: connect: connection refused" start-of-body= Sep 09 14:37:34.741674 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:34.741569 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" podUID="96887fc2-e8bc-4ae0-b45d-96f012fbcf07" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.11:8443/readyz\": dial tcp 10.134.0.11:8443: connect: connection refused" Sep 09 14:37:34.742450 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:34.742417 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" event={"ID":"ba32ff86-2186-4f15-84f4-8b2c63867723","Type":"ContainerStarted","Data":"537fd0646f65e0da1bd9a55ba2dba3eafc992b52de7f7e2e2e165dee10b94620"} Sep 09 14:37:34.764697 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:34.764651 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" podStartSLOduration=1.312509258 podStartE2EDuration="3.764637886s" totalImagesPullingTime="2.452128628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:34.763529652 +0000 UTC m=+254.430335699" watchObservedRunningTime="2026-09-09 14:37:34.764637886 +0000 UTC m=+254.431443931" Sep 09 14:37:34.785553 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:34.785511 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" podStartSLOduration=1.495467539 podStartE2EDuration="3.785495126s" totalImagesPullingTime="2.290027587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:34.784592769 +0000 UTC m=+254.451398815" watchObservedRunningTime="2026-09-09 14:37:34.785495126 +0000 UTC m=+254.452301172" Sep 09 14:37:35.271798 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.271751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:35.271994 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.271941 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:37:35.271994 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.271966 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b7bc7d5-5hcss: secret "image-registry-tls" not found Sep 09 14:37:35.272107 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.272041 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls podName:2ee8a629-5cfe-4f0a-836b-725b2a469dee nodeName:}" failed. No retries permitted until 2026-09-09 14:37:39.272024016 +0000 UTC m=+258.938830040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls") pod "image-registry-574b7bc7d5-5hcss" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee") : secret "image-registry-tls" not found Sep 09 14:37:35.372990 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.372963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:35.373075 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.373046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:35.373115 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.373079 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:35.373168 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.373114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:35.373168 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.373121 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 14:37:35.373264 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.373180 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:39.373166502 +0000 UTC m=+259.039972526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : secret "router-metrics-certs-default" not found Sep 09 14:37:35.373264 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.373201 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:35.373264 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.373219 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:39.373200949 +0000 UTC m=+259.040006991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : configmap references non-existent config key: service-ca.crt Sep 09 14:37:35.373264 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.373242 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 14:37:35.373264 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.373255 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls podName:52d3841d-c086-4a5a-99dd-6f98adffbfd9 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:39.373238172 +0000 UTC m=+259.040044212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9tkzv" (UID: "52d3841d-c086-4a5a-99dd-6f98adffbfd9") : secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:35.373556 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.373288 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls podName:e0ce45bb-2ee5-4e19-92ef-134ca514e125 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:39.373274982 +0000 UTC m=+259.040081011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-qfvdv" (UID: "e0ce45bb-2ee5-4e19-92ef-134ca514e125") : secret "samples-operator-tls" not found Sep 09 14:37:35.665244 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.665168 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6hltd"] Sep 09 14:37:35.668497 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.668474 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.671424 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.671402 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 14:37:35.671636 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.671620 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 14:37:35.671721 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.671665 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 14:37:35.672312 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.672296 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jw79l\"" Sep 09 14:37:35.672694 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.672677 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 14:37:35.682466 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.682446 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6hltd"] Sep 09 14:37:35.746668 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.746632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289" event={"ID":"f739c355-396f-4032-86b7-3ad0ccc92c48","Type":"ContainerStarted","Data":"c9da56d1b3cc138581ba1a736048a4ab50863f383c712b5451015bb38a56d1e8"} Sep 09 14:37:35.748131 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.748112 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/0.log" Sep 09 14:37:35.748227 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.748152 2578 generic.go:356] "Generic (PLEG): container finished" podID="96887fc2-e8bc-4ae0-b45d-96f012fbcf07" containerID="56dd72d8a5a9ba071c852a58ed0463aac9352fd1f5dba48796a48ac8c2328777" exitCode=255 Sep 09 14:37:35.748227 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.748215 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" event={"ID":"96887fc2-e8bc-4ae0-b45d-96f012fbcf07","Type":"ContainerDied","Data":"56dd72d8a5a9ba071c852a58ed0463aac9352fd1f5dba48796a48ac8c2328777"} Sep 09 14:37:35.748469 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.748456 2578 scope.go:117] "RemoveContainer" containerID="56dd72d8a5a9ba071c852a58ed0463aac9352fd1f5dba48796a48ac8c2328777" Sep 09 14:37:35.769363 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.769309 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-gj289" podStartSLOduration=1.6951307899999999 podStartE2EDuration="4.769292666s" totalImagesPullingTime="3.074161876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:35.767829869 +0000 UTC m=+255.434635906" watchObservedRunningTime="2026-09-09 14:37:35.769292666 +0000 UTC m=+255.436098713" Sep 09 14:37:35.775756 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.775737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a70e9268-21ad-48cc-b7ef-30a315996603-crio-socket\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.775839 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.775779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70e9268-21ad-48cc-b7ef-30a315996603-data-volume\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.775839 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.775796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.775939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.775881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.775939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.775930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2zz\" (UniqueName: \"kubernetes.io/projected/a70e9268-21ad-48cc-b7ef-30a315996603-kube-api-access-rr2zz\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.876809 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.876786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.876897 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.876826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2zz\" (UniqueName: \"kubernetes.io/projected/a70e9268-21ad-48cc-b7ef-30a315996603-kube-api-access-rr2zz\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.876936 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.876900 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:35.876979 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.876967 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:36.376947861 +0000 UTC m=+256.043753899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : configmap "kube-rbac-proxy" not found Sep 09 14:37:35.877207 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.877184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a70e9268-21ad-48cc-b7ef-30a315996603-crio-socket\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.877296 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.877271 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a70e9268-21ad-48cc-b7ef-30a315996603-crio-socket\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.877383 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.877304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70e9268-21ad-48cc-b7ef-30a315996603-data-volume\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.877383 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.877336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.877488 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.877450 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:35.877540 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:35.877505 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:36.377485851 +0000 UTC m=+256.044291889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:35.877602 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.877586 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70e9268-21ad-48cc-b7ef-30a315996603-data-volume\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:35.887364 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:35.887331 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2zz\" (UniqueName: \"kubernetes.io/projected/a70e9268-21ad-48cc-b7ef-30a315996603-kube-api-access-rr2zz\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:36.382250 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:36.382217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:36.382427 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:36.382318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:36.382427 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:36.382364 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:36.382427 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:36.382415 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:36.382537 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:36.382432 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:37.38241696 +0000 UTC m=+257.049223008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : configmap "kube-rbac-proxy" not found Sep 09 14:37:36.382537 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:36.382493 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:37.382481887 +0000 UTC m=+257.049287911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:36.752504 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:36.752477 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:37:36.752872 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:36.752859 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/0.log" Sep 09 14:37:36.752925 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:36.752891 2578 generic.go:356] "Generic (PLEG): container finished" podID="96887fc2-e8bc-4ae0-b45d-96f012fbcf07" containerID="c1b7605deed5f271aa159d6d435bdeaf2a89c13081bd5668d571395e6f261022" exitCode=255 Sep 09 14:37:36.752984 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:36.752961 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" event={"ID":"96887fc2-e8bc-4ae0-b45d-96f012fbcf07","Type":"ContainerDied","Data":"c1b7605deed5f271aa159d6d435bdeaf2a89c13081bd5668d571395e6f261022"} Sep 09 14:37:36.753030 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:36.753018 2578 scope.go:117] "RemoveContainer" containerID="56dd72d8a5a9ba071c852a58ed0463aac9352fd1f5dba48796a48ac8c2328777" Sep 09 14:37:36.753208 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:36.753194 2578 scope.go:117] "RemoveContainer" containerID="c1b7605deed5f271aa159d6d435bdeaf2a89c13081bd5668d571395e6f261022" Sep 09 14:37:36.753414 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:36.753392 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-n7g7g_openshift-console-operator(96887fc2-e8bc-4ae0-b45d-96f012fbcf07)\"" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" podUID="96887fc2-e8bc-4ae0-b45d-96f012fbcf07" Sep 09 14:37:37.390750 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:37.390710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:37.390983 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:37.390831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:37.390983 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:37.390840 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:37.390983 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:37.390906 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:39.39089019 +0000 UTC m=+259.057696214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : configmap "kube-rbac-proxy" not found Sep 09 14:37:37.390983 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:37.390928 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:37.390983 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:37.390967 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:39.390955525 +0000 UTC m=+259.057761549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:37.757444 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:37.757416 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:37:37.757808 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:37.757744 2578 scope.go:117] "RemoveContainer" containerID="c1b7605deed5f271aa159d6d435bdeaf2a89c13081bd5668d571395e6f261022" Sep 09 14:37:37.757908 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:37.757887 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-n7g7g_openshift-console-operator(96887fc2-e8bc-4ae0-b45d-96f012fbcf07)\"" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" podUID="96887fc2-e8bc-4ae0-b45d-96f012fbcf07" Sep 09 14:37:37.809686 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:37.809661 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qhwvw_fffaf1ff-bc62-4ce6-a5ca-a45d536a4217/dns-node-resolver/0.log" Sep 09 14:37:38.411429 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:38.411392 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4v9k9_8b9fd951-66f4-459d-966d-6565d97cec02/node-ca/0.log" Sep 09 14:37:39.306420 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:39.306380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:39.306824 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.306517 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:37:39.306824 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.306533 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b7bc7d5-5hcss: secret "image-registry-tls" not found Sep 09 14:37:39.306824 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.306586 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls podName:2ee8a629-5cfe-4f0a-836b-725b2a469dee nodeName:}" failed. No retries permitted until 2026-09-09 14:37:47.306570385 +0000 UTC m=+266.973376409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls") pod "image-registry-574b7bc7d5-5hcss" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee") : secret "image-registry-tls" not found Sep 09 14:37:39.407297 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:39.407260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:39.407297 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:39.407296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:39.407327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407397 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407429 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407442 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:43.407429322 +0000 UTC m=+263.074235346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : configmap "kube-rbac-proxy" not found Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:39.407463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407480 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:47.407467972 +0000 UTC m=+267.074273995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : secret "router-metrics-certs-default" not found Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407434 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:39.407520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407525 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:43.407512617 +0000 UTC m=+263.074318642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:39.407564 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407549 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:37:47.407537206 +0000 UTC m=+267.074343230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : configmap references non-existent config key: service-ca.crt Sep 09 14:37:39.407923 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407585 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:39.407923 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407622 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls podName:52d3841d-c086-4a5a-99dd-6f98adffbfd9 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:47.40760985 +0000 UTC m=+267.074415876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9tkzv" (UID: "52d3841d-c086-4a5a-99dd-6f98adffbfd9") : secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:39.407923 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407626 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 14:37:39.407923 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:39.407581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:39.407923 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:39.407649 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls podName:e0ce45bb-2ee5-4e19-92ef-134ca514e125 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:47.407642052 +0000 UTC m=+267.074448077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-qfvdv" (UID: "e0ce45bb-2ee5-4e19-92ef-134ca514e125") : secret "samples-operator-tls" not found Sep 09 14:37:39.610505 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:39.610409 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5ntb8_9ef9739c-da6d-450b-be72-a2d7c60b9027/serve-healthcheck-canary/0.log" Sep 09 14:37:41.913581 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:41.913497 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:41.914023 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:41.913933 2578 scope.go:117] "RemoveContainer" containerID="c1b7605deed5f271aa159d6d435bdeaf2a89c13081bd5668d571395e6f261022" Sep 09 14:37:41.917872 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:41.914107 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-n7g7g_openshift-console-operator(96887fc2-e8bc-4ae0-b45d-96f012fbcf07)\"" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" podUID="96887fc2-e8bc-4ae0-b45d-96f012fbcf07" Sep 09 14:37:43.440341 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:43.440253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:43.440773 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:43.440392 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 14:37:43.440773 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:43.440415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:43.440773 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:43.440456 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:51.440441967 +0000 UTC m=+271.107247997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : configmap "kube-rbac-proxy" not found Sep 09 14:37:43.440773 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:43.440526 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:43.440773 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:43.440581 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:37:51.440565871 +0000 UTC m=+271.107371898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:44.740676 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:44.740640 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:44.741048 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:44.741007 2578 scope.go:117] "RemoveContainer" containerID="c1b7605deed5f271aa159d6d435bdeaf2a89c13081bd5668d571395e6f261022" Sep 09 14:37:44.741172 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:44.741151 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-n7g7g_openshift-console-operator(96887fc2-e8bc-4ae0-b45d-96f012fbcf07)\"" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" podUID="96887fc2-e8bc-4ae0-b45d-96f012fbcf07" Sep 09 14:37:47.373058 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:47.373021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls\") pod \"image-registry-574b7bc7d5-5hcss\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:47.373417 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:47.373140 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 14:37:47.373417 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:47.373152 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b7bc7d5-5hcss: secret "image-registry-tls" not found Sep 09 14:37:47.373417 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:47.373200 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls podName:2ee8a629-5cfe-4f0a-836b-725b2a469dee nodeName:}" failed. No retries permitted until 2026-09-09 14:38:03.373185981 +0000 UTC m=+283.039992005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls") pod "image-registry-574b7bc7d5-5hcss" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee") : secret "image-registry-tls" not found Sep 09 14:37:47.474133 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:47.474104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:47.474236 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:47.474139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:37:47.474236 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:47.474175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:47.474307 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:47.474247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:47.474307 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:47.474272 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle podName:712c3b38-ef28-4237-8de9-1d9a6058866c nodeName:}" failed. No retries permitted until 2026-09-09 14:38:03.474253908 +0000 UTC m=+283.141059936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle") pod "router-default-594c74b48d-2brjb" (UID: "712c3b38-ef28-4237-8de9-1d9a6058866c") : configmap references non-existent config key: service-ca.crt Sep 09 14:37:47.474406 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:47.474270 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:47.474444 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:47.474432 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls podName:52d3841d-c086-4a5a-99dd-6f98adffbfd9 nodeName:}" failed. No retries permitted until 2026-09-09 14:38:03.474407474 +0000 UTC m=+283.141213515 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9tkzv" (UID: "52d3841d-c086-4a5a-99dd-6f98adffbfd9") : secret "cluster-monitoring-operator-tls" not found Sep 09 14:37:47.476620 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:47.476591 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712c3b38-ef28-4237-8de9-1d9a6058866c-metrics-certs\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:37:47.476708 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:47.476669 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0ce45bb-2ee5-4e19-92ef-134ca514e125-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-qfvdv\" (UID: \"e0ce45bb-2ee5-4e19-92ef-134ca514e125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:47.579836 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:47.579811 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" Sep 09 14:37:47.849101 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:47.849071 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv"] Sep 09 14:37:48.792050 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:48.792014 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" event={"ID":"e0ce45bb-2ee5-4e19-92ef-134ca514e125","Type":"ContainerStarted","Data":"aeea2bdadf5d7165fab60d19c31f2b92188d8709b731a1a800c333437c11d3dc"} Sep 09 14:37:50.799847 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:50.799818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" event={"ID":"e0ce45bb-2ee5-4e19-92ef-134ca514e125","Type":"ContainerStarted","Data":"70fc82d9529ff28328a0c460ae3f183c561780e5c71c25cfb3629483af1d7e29"} Sep 09 14:37:50.800220 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:50.799865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" event={"ID":"e0ce45bb-2ee5-4e19-92ef-134ca514e125","Type":"ContainerStarted","Data":"05f6736eaaa029e001730ee0bf4f3008f6391649cd123acbaafd0f23e6a0eed8"} Sep 09 14:37:50.821189 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:50.821140 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-qfvdv" podStartSLOduration=17.705009755 podStartE2EDuration="19.821124022s" totalImagesPullingTime="2.116114267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:37:50.820177017 +0000 UTC m=+270.486983086" watchObservedRunningTime="2026-09-09 14:37:50.821124022 +0000 UTC m=+270.487930067" Sep 09 14:37:51.509114 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:51.509072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:51.509292 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:51.509130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:51.509292 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:51.509236 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 14:37:51.509388 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:51.509295 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls podName:a70e9268-21ad-48cc-b7ef-30a315996603 nodeName:}" failed. No retries permitted until 2026-09-09 14:38:07.509281052 +0000 UTC m=+287.176087076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hltd" (UID: "a70e9268-21ad-48cc-b7ef-30a315996603") : secret "insights-runtime-extractor-tls" not found Sep 09 14:37:51.509688 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:51.509667 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a70e9268-21ad-48cc-b7ef-30a315996603-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:37:58.534229 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.534195 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-574b7bc7d5-5hcss"] Sep 09 14:37:58.534615 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:37:58.534428 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" podUID="2ee8a629-5cfe-4f0a-836b-725b2a469dee" Sep 09 14:37:58.822868 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.822793 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:58.829044 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.829020 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:58.861058 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861037 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-certificates\") pod \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " Sep 09 14:37:58.861195 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861073 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-installation-pull-secrets\") pod \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " Sep 09 14:37:58.861195 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861102 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-image-registry-private-configuration\") pod \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " Sep 09 14:37:58.861195 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861132 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55vsh\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-kube-api-access-55vsh\") pod \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " Sep 09 14:37:58.861195 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861153 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-trusted-ca\") pod \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " Sep 09 14:37:58.861195 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861180 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ee8a629-5cfe-4f0a-836b-725b2a469dee-ca-trust-extracted\") pod \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " Sep 09 14:37:58.861474 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861202 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-bound-sa-token\") pod \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\" (UID: \"2ee8a629-5cfe-4f0a-836b-725b2a469dee\") " Sep 09 14:37:58.861474 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861434 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2ee8a629-5cfe-4f0a-836b-725b2a469dee" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:37:58.861590 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861562 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ee8a629-5cfe-4f0a-836b-725b2a469dee-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2ee8a629-5cfe-4f0a-836b-725b2a469dee" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:37:58.861664 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.861640 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2ee8a629-5cfe-4f0a-836b-725b2a469dee" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:37:58.863432 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.863398 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2ee8a629-5cfe-4f0a-836b-725b2a469dee" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:37:58.863545 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.863521 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2ee8a629-5cfe-4f0a-836b-725b2a469dee" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:37:58.863743 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.863717 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2ee8a629-5cfe-4f0a-836b-725b2a469dee" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:37:58.863795 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.863771 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-kube-api-access-55vsh" (OuterVolumeSpecName: "kube-api-access-55vsh") pod "2ee8a629-5cfe-4f0a-836b-725b2a469dee" (UID: "2ee8a629-5cfe-4f0a-836b-725b2a469dee"). InnerVolumeSpecName "kube-api-access-55vsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:37:58.962303 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.962265 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-installation-pull-secrets\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:37:58.962303 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.962298 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ee8a629-5cfe-4f0a-836b-725b2a469dee-image-registry-private-configuration\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:37:58.962303 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.962309 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55vsh\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-kube-api-access-55vsh\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:37:58.962303 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.962319 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-trusted-ca\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:37:58.962566 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.962328 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ee8a629-5cfe-4f0a-836b-725b2a469dee-ca-trust-extracted\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:37:58.962566 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.962336 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-bound-sa-token\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:37:58.962566 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.962364 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-certificates\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:37:58.976612 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:58.976589 2578 scope.go:117] "RemoveContainer" containerID="c1b7605deed5f271aa159d6d435bdeaf2a89c13081bd5668d571395e6f261022" Sep 09 14:37:59.828019 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:59.827992 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:37:59.828401 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:59.828083 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574b7bc7d5-5hcss" Sep 09 14:37:59.828401 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:59.828110 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" event={"ID":"96887fc2-e8bc-4ae0-b45d-96f012fbcf07","Type":"ContainerStarted","Data":"b469a1cc7231135044a5ded0c07afb7eda510593b9191ef316989e3617d77438"} Sep 09 14:37:59.828505 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:59.828442 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:59.833190 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:59.833169 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-n7g7g" Sep 09 14:37:59.881851 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:59.881827 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-574b7bc7d5-5hcss"] Sep 09 14:37:59.886869 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:59.886840 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-574b7bc7d5-5hcss"] Sep 09 14:37:59.970105 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:37:59.970079 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ee8a629-5cfe-4f0a-836b-725b2a469dee-registry-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:38:00.979725 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:00.979685 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee8a629-5cfe-4f0a-836b-725b2a469dee" path="/var/lib/kubelet/pods/2ee8a629-5cfe-4f0a-836b-725b2a469dee/volumes" Sep 09 14:38:01.495867 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:38:01.495828 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9v8sj" podUID="e2994b71-2a39-42b9-bd8c-1b55c03259b0" Sep 09 14:38:01.834054 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:01.833966 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9v8sj" Sep 09 14:38:03.494606 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:03.494562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:38:03.494606 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:03.494604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:38:03.495304 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:03.495282 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/712c3b38-ef28-4237-8de9-1d9a6058866c-service-ca-bundle\") pod \"router-default-594c74b48d-2brjb\" (UID: \"712c3b38-ef28-4237-8de9-1d9a6058866c\") " pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:38:03.496978 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:03.496953 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52d3841d-c086-4a5a-99dd-6f98adffbfd9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9tkzv\" (UID: \"52d3841d-c086-4a5a-99dd-6f98adffbfd9\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:38:03.703613 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:03.703572 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" Sep 09 14:38:03.717277 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:03.717245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:38:03.978472 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:03.978445 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv"] Sep 09 14:38:03.980384 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:03.980338 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d3841d_c086_4a5a_99dd_6f98adffbfd9.slice/crio-e283326cefbd1ba7d4a05cf6b46ea4cba7e0556957a64648319073bd1c2cb3b1 WatchSource:0}: Error finding container e283326cefbd1ba7d4a05cf6b46ea4cba7e0556957a64648319073bd1c2cb3b1: Status 404 returned error can't find the container with id e283326cefbd1ba7d4a05cf6b46ea4cba7e0556957a64648319073bd1c2cb3b1 Sep 09 14:38:03.999760 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:03.999735 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-594c74b48d-2brjb"] Sep 09 14:38:04.002218 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:04.002196 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712c3b38_ef28_4237_8de9_1d9a6058866c.slice/crio-1df0d9cd1db5effe5bdff8c1e7a05dd1babd1be8b9e6dcca5da831a1ae052eec WatchSource:0}: Error finding container 1df0d9cd1db5effe5bdff8c1e7a05dd1babd1be8b9e6dcca5da831a1ae052eec: Status 404 returned error can't find the container with id 1df0d9cd1db5effe5bdff8c1e7a05dd1babd1be8b9e6dcca5da831a1ae052eec Sep 09 14:38:04.602842 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:04.602806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:38:04.604981 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:04.604960 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2994b71-2a39-42b9-bd8c-1b55c03259b0-metrics-tls\") pod \"dns-default-9v8sj\" (UID: \"e2994b71-2a39-42b9-bd8c-1b55c03259b0\") " pod="openshift-dns/dns-default-9v8sj" Sep 09 14:38:04.837359 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:04.837323 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btvxq\"" Sep 09 14:38:04.844813 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:04.844779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9v8sj" Sep 09 14:38:04.845457 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:04.845187 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-594c74b48d-2brjb" event={"ID":"712c3b38-ef28-4237-8de9-1d9a6058866c","Type":"ContainerStarted","Data":"8bedf08ba1a3bb9ef1ba7a1fc5db26581f65a87561df1accdbbd8903bf23d817"} Sep 09 14:38:04.845457 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:04.845240 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-594c74b48d-2brjb" event={"ID":"712c3b38-ef28-4237-8de9-1d9a6058866c","Type":"ContainerStarted","Data":"1df0d9cd1db5effe5bdff8c1e7a05dd1babd1be8b9e6dcca5da831a1ae052eec"} Sep 09 14:38:04.846821 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:04.846590 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" event={"ID":"52d3841d-c086-4a5a-99dd-6f98adffbfd9","Type":"ContainerStarted","Data":"e283326cefbd1ba7d4a05cf6b46ea4cba7e0556957a64648319073bd1c2cb3b1"} Sep 09 14:38:04.867376 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:04.866637 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-594c74b48d-2brjb" podStartSLOduration=33.866618279 podStartE2EDuration="33.866618279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:04.865455182 +0000 UTC m=+284.532261311" watchObservedRunningTime="2026-09-09 14:38:04.866618279 +0000 UTC m=+284.533424326" Sep 09 14:38:05.128065 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:05.127982 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9v8sj"] Sep 09 14:38:05.131273 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:05.131249 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2994b71_2a39_42b9_bd8c_1b55c03259b0.slice/crio-b1776a329965dbba546024cddf9be84d94abc4ccbff4a9c58696f98600e06131 WatchSource:0}: Error finding container b1776a329965dbba546024cddf9be84d94abc4ccbff4a9c58696f98600e06131: Status 404 returned error can't find the container with id b1776a329965dbba546024cddf9be84d94abc4ccbff4a9c58696f98600e06131 Sep 09 14:38:05.718178 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:05.718154 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:38:05.720980 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:05.720959 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:38:05.851762 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:05.851725 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9v8sj" event={"ID":"e2994b71-2a39-42b9-bd8c-1b55c03259b0","Type":"ContainerStarted","Data":"b1776a329965dbba546024cddf9be84d94abc4ccbff4a9c58696f98600e06131"} Sep 09 14:38:05.853634 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:05.853596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" event={"ID":"52d3841d-c086-4a5a-99dd-6f98adffbfd9","Type":"ContainerStarted","Data":"b388c0d25adcb755eb2981b207910194b5708d490a2047985525d88401dcdada"} Sep 09 14:38:05.853764 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:05.853653 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:38:05.855021 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:05.855001 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-594c74b48d-2brjb" Sep 09 14:38:05.869572 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:05.869520 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9tkzv" podStartSLOduration=33.198630112 podStartE2EDuration="34.869502995s" totalImagesPullingTime="1.670872883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:05.868923136 +0000 UTC m=+285.535729182" watchObservedRunningTime="2026-09-09 14:38:05.869502995 +0000 UTC m=+285.536309046" Sep 09 14:38:06.160180 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:06.160150 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l"] Sep 09 14:38:06.162312 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:06.162289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" Sep 09 14:38:06.164607 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:06.164586 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 14:38:06.164736 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:06.164629 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-77v99\"" Sep 09 14:38:06.172882 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:06.172862 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l"] Sep 09 14:38:06.219923 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:06.219889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55b8f528-bf00-4509-bb5a-7ef7083dfb0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kb94l\" (UID: \"55b8f528-bf00-4509-bb5a-7ef7083dfb0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" Sep 09 14:38:06.321237 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:06.321204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55b8f528-bf00-4509-bb5a-7ef7083dfb0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kb94l\" (UID: \"55b8f528-bf00-4509-bb5a-7ef7083dfb0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" Sep 09 14:38:06.321406 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:38:06.321338 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 09 14:38:06.321472 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:38:06.321424 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55b8f528-bf00-4509-bb5a-7ef7083dfb0a-tls-certificates podName:55b8f528-bf00-4509-bb5a-7ef7083dfb0a nodeName:}" failed. No retries permitted until 2026-09-09 14:38:06.82140072 +0000 UTC m=+286.488206759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/55b8f528-bf00-4509-bb5a-7ef7083dfb0a-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-kb94l" (UID: "55b8f528-bf00-4509-bb5a-7ef7083dfb0a") : secret "prometheus-operator-admission-webhook-tls" not found Sep 09 14:38:06.825400 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:06.825369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55b8f528-bf00-4509-bb5a-7ef7083dfb0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kb94l\" (UID: \"55b8f528-bf00-4509-bb5a-7ef7083dfb0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" Sep 09 14:38:06.827946 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:06.827908 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/55b8f528-bf00-4509-bb5a-7ef7083dfb0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kb94l\" (UID: \"55b8f528-bf00-4509-bb5a-7ef7083dfb0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" Sep 09 14:38:07.072651 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:07.072569 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" Sep 09 14:38:07.338042 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:07.337964 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l"] Sep 09 14:38:07.340653 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:07.340630 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b8f528_bf00_4509_bb5a_7ef7083dfb0a.slice/crio-968486e1575dd1ad77d91f4c5e30eb3bb53b496248ade5b8449dc8f8edf40b04 WatchSource:0}: Error finding container 968486e1575dd1ad77d91f4c5e30eb3bb53b496248ade5b8449dc8f8edf40b04: Status 404 returned error can't find the container with id 968486e1575dd1ad77d91f4c5e30eb3bb53b496248ade5b8449dc8f8edf40b04 Sep 09 14:38:07.530339 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:07.530298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:38:07.532558 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:07.532528 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a70e9268-21ad-48cc-b7ef-30a315996603-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hltd\" (UID: \"a70e9268-21ad-48cc-b7ef-30a315996603\") " pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:38:07.777486 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:07.777457 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6hltd" Sep 09 14:38:07.863763 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:07.863729 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" event={"ID":"55b8f528-bf00-4509-bb5a-7ef7083dfb0a","Type":"ContainerStarted","Data":"968486e1575dd1ad77d91f4c5e30eb3bb53b496248ade5b8449dc8f8edf40b04"} Sep 09 14:38:07.865518 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:07.865494 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9v8sj" event={"ID":"e2994b71-2a39-42b9-bd8c-1b55c03259b0","Type":"ContainerStarted","Data":"04098f749f12e911d736bc16ae7d0ea3cbba6d974d57813bc7c390cee95befd3"} Sep 09 14:38:07.865626 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:07.865536 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9v8sj" event={"ID":"e2994b71-2a39-42b9-bd8c-1b55c03259b0","Type":"ContainerStarted","Data":"255ac2622cc553907c759485751119c0b64f24d5097501fadf3e5604597cec5a"} Sep 09 14:38:07.881174 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:07.881127 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9v8sj" podStartSLOduration=252.235852451 podStartE2EDuration="4m13.881113013s" totalImagesPullingTime="1.645260562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:07.880231085 +0000 UTC m=+287.547037131" watchObservedRunningTime="2026-09-09 14:38:07.881113013 +0000 UTC m=+287.547919112" Sep 09 14:38:08.048478 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:08.048450 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6hltd"] Sep 09 14:38:08.050951 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:08.050924 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70e9268_21ad_48cc_b7ef_30a315996603.slice/crio-6207add976ecb0d832e04ad4d3e88e53d79337d4e44d20c6f3fb4bf03f25e4d7 WatchSource:0}: Error finding container 6207add976ecb0d832e04ad4d3e88e53d79337d4e44d20c6f3fb4bf03f25e4d7: Status 404 returned error can't find the container with id 6207add976ecb0d832e04ad4d3e88e53d79337d4e44d20c6f3fb4bf03f25e4d7 Sep 09 14:38:08.871972 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:08.871893 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6hltd" event={"ID":"a70e9268-21ad-48cc-b7ef-30a315996603","Type":"ContainerStarted","Data":"d19a08d1c69617f98dc3edfa1dcdca03ef8f7913280ce1afdc121a721cea0219"} Sep 09 14:38:08.872391 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:08.871970 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6hltd" event={"ID":"a70e9268-21ad-48cc-b7ef-30a315996603","Type":"ContainerStarted","Data":"6207add976ecb0d832e04ad4d3e88e53d79337d4e44d20c6f3fb4bf03f25e4d7"} Sep 09 14:38:08.873291 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:08.873263 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" event={"ID":"55b8f528-bf00-4509-bb5a-7ef7083dfb0a","Type":"ContainerStarted","Data":"e4eea0fe060012deba34fd921fc993c131ac56440bc7af4cff724972e8593af5"} Sep 09 14:38:08.873691 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:08.873675 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" Sep 09 14:38:08.873790 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:08.873706 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9v8sj" Sep 09 14:38:08.878382 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:08.878360 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" Sep 09 14:38:08.889783 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:08.889742 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kb94l" podStartSLOduration=1.688392276 podStartE2EDuration="2.889728378s" totalImagesPullingTime="1.201336102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:08.888803995 +0000 UTC m=+288.555610041" watchObservedRunningTime="2026-09-09 14:38:08.889728378 +0000 UTC m=+288.556534455" Sep 09 14:38:09.221156 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.221125 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xzq54"] Sep 09 14:38:09.224406 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.224391 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.227001 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.226976 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 09 14:38:09.227001 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.226995 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sld69\"" Sep 09 14:38:09.227166 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.226988 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 09 14:38:09.227166 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.227036 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 14:38:09.234430 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.234413 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xzq54"] Sep 09 14:38:09.245525 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.245493 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr59n\" (UniqueName: \"kubernetes.io/projected/6e992a04-0505-4355-aa11-b1038a72882f-kube-api-access-rr59n\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.245615 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.245535 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e992a04-0505-4355-aa11-b1038a72882f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.245615 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.245588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e992a04-0505-4355-aa11-b1038a72882f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.245724 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.245632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e992a04-0505-4355-aa11-b1038a72882f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.346900 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.346850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr59n\" (UniqueName: \"kubernetes.io/projected/6e992a04-0505-4355-aa11-b1038a72882f-kube-api-access-rr59n\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.347082 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.346918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e992a04-0505-4355-aa11-b1038a72882f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.347082 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.346949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e992a04-0505-4355-aa11-b1038a72882f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.347082 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.346991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e992a04-0505-4355-aa11-b1038a72882f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.347082 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:38:09.347067 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 09 14:38:09.347286 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:38:09.347141 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e992a04-0505-4355-aa11-b1038a72882f-prometheus-operator-tls podName:6e992a04-0505-4355-aa11-b1038a72882f nodeName:}" failed. No retries permitted until 2026-09-09 14:38:09.847119214 +0000 UTC m=+289.513925238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6e992a04-0505-4355-aa11-b1038a72882f-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-xzq54" (UID: "6e992a04-0505-4355-aa11-b1038a72882f") : secret "prometheus-operator-tls" not found Sep 09 14:38:09.347698 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.347678 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e992a04-0505-4355-aa11-b1038a72882f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.349181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.349164 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e992a04-0505-4355-aa11-b1038a72882f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.354863 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.354840 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr59n\" (UniqueName: \"kubernetes.io/projected/6e992a04-0505-4355-aa11-b1038a72882f-kube-api-access-rr59n\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.850956 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.850924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e992a04-0505-4355-aa11-b1038a72882f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.853507 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.853481 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e992a04-0505-4355-aa11-b1038a72882f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-xzq54\" (UID: \"6e992a04-0505-4355-aa11-b1038a72882f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:09.880412 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:09.880378 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6hltd" event={"ID":"a70e9268-21ad-48cc-b7ef-30a315996603","Type":"ContainerStarted","Data":"17ad0e14f172249d90d65afa8897ed45e95e0a450d0e4297c208ab75c8aa14a6"} Sep 09 14:38:10.133182 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:10.133099 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" Sep 09 14:38:10.409884 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:10.409720 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-xzq54"] Sep 09 14:38:10.627275 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:10.627245 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e992a04_0505_4355_aa11_b1038a72882f.slice/crio-5ba8e12f5f41beeaff4c6580839426ce06db4a6fe6ca515d280c209e581705bc WatchSource:0}: Error finding container 5ba8e12f5f41beeaff4c6580839426ce06db4a6fe6ca515d280c209e581705bc: Status 404 returned error can't find the container with id 5ba8e12f5f41beeaff4c6580839426ce06db4a6fe6ca515d280c209e581705bc Sep 09 14:38:10.885612 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:10.885574 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6hltd" event={"ID":"a70e9268-21ad-48cc-b7ef-30a315996603","Type":"ContainerStarted","Data":"5634ab317d6caaeb423ba6ac27a8bf2f4a04deb2be2f28b4dc3ee179a5419f76"} Sep 09 14:38:10.886885 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:10.886860 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" event={"ID":"6e992a04-0505-4355-aa11-b1038a72882f","Type":"ContainerStarted","Data":"5ba8e12f5f41beeaff4c6580839426ce06db4a6fe6ca515d280c209e581705bc"} Sep 09 14:38:10.908930 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:10.908888 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6hltd" podStartSLOduration=33.404934106 podStartE2EDuration="35.908872801s" totalImagesPullingTime="2.503938695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 14:37:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:10.907554129 +0000 UTC m=+290.574360174" watchObservedRunningTime="2026-09-09 14:38:10.908872801 +0000 UTC m=+290.575678847" Sep 09 14:38:12.587221 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.587143 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb66dbb94-9gzlf"] Sep 09 14:38:12.590132 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.590113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.592870 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.592627 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 14:38:12.592870 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.592857 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 14:38:12.593415 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.593394 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 14:38:12.593659 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.593633 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 14:38:12.593764 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.593727 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 14:38:12.593764 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.593727 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 14:38:12.593853 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.593785 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 14:38:12.594011 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.593999 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wcftc\"" Sep 09 14:38:12.601110 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.601090 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb66dbb94-9gzlf"] Sep 09 14:38:12.672629 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.672588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-oauth-serving-cert\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.672784 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.672693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqb68\" (UniqueName: \"kubernetes.io/projected/c4dda96e-8c13-4b84-9c75-c136133dc0bc-kube-api-access-sqb68\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.672784 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.672734 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-service-ca\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.672784 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.672761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-config\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.672784 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.672783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-serving-cert\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.672912 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.672816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-oauth-config\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.774136 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.774093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-oauth-serving-cert\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.774328 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.774155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqb68\" (UniqueName: \"kubernetes.io/projected/c4dda96e-8c13-4b84-9c75-c136133dc0bc-kube-api-access-sqb68\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.774328 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.774179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-service-ca\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.774328 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.774199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-config\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.774328 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.774219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-serving-cert\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.774328 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.774235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-oauth-config\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.774866 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.774845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-oauth-serving-cert\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.774939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.774920 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-config\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.774985 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.774954 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-service-ca\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.776585 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.776558 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-oauth-config\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.776706 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.776690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-serving-cert\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.782406 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.782387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqb68\" (UniqueName: \"kubernetes.io/projected/c4dda96e-8c13-4b84-9c75-c136133dc0bc-kube-api-access-sqb68\") pod \"console-bb66dbb94-9gzlf\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.899006 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.898923 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:12.899158 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.899134 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" event={"ID":"6e992a04-0505-4355-aa11-b1038a72882f","Type":"ContainerStarted","Data":"3389cef8b7fc11441a05278d5834eb2482f1232781b5916d15227bd8f4718944"} Sep 09 14:38:12.899218 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.899207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" event={"ID":"6e992a04-0505-4355-aa11-b1038a72882f","Type":"ContainerStarted","Data":"b66da15ed9b198f82335098fa761ce83e1ed4379d263dbeb028cfa903e0e0f3a"} Sep 09 14:38:12.921211 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:12.921167 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-xzq54" podStartSLOduration=2.263763765 podStartE2EDuration="3.921151162s" totalImagesPullingTime="1.657387397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:12.919956739 +0000 UTC m=+292.586762800" watchObservedRunningTime="2026-09-09 14:38:12.921151162 +0000 UTC m=+292.587957207" Sep 09 14:38:13.171836 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:13.171765 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb66dbb94-9gzlf"] Sep 09 14:38:13.174817 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:13.174787 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4dda96e_8c13_4b84_9c75_c136133dc0bc.slice/crio-b1f4848d0abe2455ccdba218ed50bcd6c3213c1868ac5fbd278a9684301f7db2 WatchSource:0}: Error finding container b1f4848d0abe2455ccdba218ed50bcd6c3213c1868ac5fbd278a9684301f7db2: Status 404 returned error can't find the container with id b1f4848d0abe2455ccdba218ed50bcd6c3213c1868ac5fbd278a9684301f7db2 Sep 09 14:38:13.905008 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:13.904964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb66dbb94-9gzlf" event={"ID":"c4dda96e-8c13-4b84-9c75-c136133dc0bc","Type":"ContainerStarted","Data":"b1f4848d0abe2455ccdba218ed50bcd6c3213c1868ac5fbd278a9684301f7db2"} Sep 09 14:38:14.601958 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.601920 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2dlbn"] Sep 09 14:38:14.605531 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.605509 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.607837 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.607816 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 14:38:14.608200 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.608180 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 14:38:14.608293 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.608241 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j5n54\"" Sep 09 14:38:14.608380 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.608239 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 14:38:14.689554 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.689530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-textfile\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.689698 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.689561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14079d71-4d00-41e3-b4f0-efab1cf10451-sys\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.689698 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.689596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-accelerators-collector-config\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.689698 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.689624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-tls\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.689698 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.689684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hxt\" (UniqueName: \"kubernetes.io/projected/14079d71-4d00-41e3-b4f0-efab1cf10451-kube-api-access-r5hxt\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.689925 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.689750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14079d71-4d00-41e3-b4f0-efab1cf10451-root\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.689925 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.689779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.689925 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.689848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-wtmp\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.689925 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.689882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14079d71-4d00-41e3-b4f0-efab1cf10451-metrics-client-ca\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791245 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-wtmp\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791430 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14079d71-4d00-41e3-b4f0-efab1cf10451-metrics-client-ca\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791507 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-textfile\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791507 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791470 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14079d71-4d00-41e3-b4f0-efab1cf10451-sys\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791507 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791430 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-wtmp\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791658 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791547 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14079d71-4d00-41e3-b4f0-efab1cf10451-sys\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791658 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-accelerators-collector-config\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791658 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-tls\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791658 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hxt\" (UniqueName: \"kubernetes.io/projected/14079d71-4d00-41e3-b4f0-efab1cf10451-kube-api-access-r5hxt\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791845 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14079d71-4d00-41e3-b4f0-efab1cf10451-root\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791845 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791845 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-textfile\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791845 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:38:14.791768 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 14:38:14.791845 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791788 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14079d71-4d00-41e3-b4f0-efab1cf10451-root\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.791845 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:38:14.791832 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-tls podName:14079d71-4d00-41e3-b4f0-efab1cf10451 nodeName:}" failed. No retries permitted until 2026-09-09 14:38:15.291811974 +0000 UTC m=+294.958617999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-tls") pod "node-exporter-2dlbn" (UID: "14079d71-4d00-41e3-b4f0-efab1cf10451") : secret "node-exporter-tls" not found Sep 09 14:38:14.792168 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.791929 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14079d71-4d00-41e3-b4f0-efab1cf10451-metrics-client-ca\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.792168 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.792078 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-accelerators-collector-config\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.794289 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.794265 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.801792 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:14.801768 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hxt\" (UniqueName: \"kubernetes.io/projected/14079d71-4d00-41e3-b4f0-efab1cf10451-kube-api-access-r5hxt\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:14.977651 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:38:14.977609 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fs9bf" podUID="c566cce4-7251-42bf-8696-f2505562d073" Sep 09 14:38:15.295131 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:15.295046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-tls\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:15.297717 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:15.297690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14079d71-4d00-41e3-b4f0-efab1cf10451-node-exporter-tls\") pod \"node-exporter-2dlbn\" (UID: \"14079d71-4d00-41e3-b4f0-efab1cf10451\") " pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:15.516257 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:15.516218 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2dlbn" Sep 09 14:38:16.033160 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:16.033133 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14079d71_4d00_41e3_b4f0_efab1cf10451.slice/crio-f6a90885867dbc3b147798efb26e03c2bd99a69077824f99c69bbdeccfe8f2a1 WatchSource:0}: Error finding container f6a90885867dbc3b147798efb26e03c2bd99a69077824f99c69bbdeccfe8f2a1: Status 404 returned error can't find the container with id f6a90885867dbc3b147798efb26e03c2bd99a69077824f99c69bbdeccfe8f2a1 Sep 09 14:38:16.917953 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:16.917909 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb66dbb94-9gzlf" event={"ID":"c4dda96e-8c13-4b84-9c75-c136133dc0bc","Type":"ContainerStarted","Data":"4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2"} Sep 09 14:38:16.919500 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:16.919289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2dlbn" event={"ID":"14079d71-4d00-41e3-b4f0-efab1cf10451","Type":"ContainerStarted","Data":"f6a90885867dbc3b147798efb26e03c2bd99a69077824f99c69bbdeccfe8f2a1"} Sep 09 14:38:16.940859 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:16.940810 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bb66dbb94-9gzlf" podStartSLOduration=2.034797476 podStartE2EDuration="4.94079099s" totalImagesPullingTime="2.905993514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:16.938641214 +0000 UTC m=+296.605447271" watchObservedRunningTime="2026-09-09 14:38:16.94079099 +0000 UTC m=+296.607597039" Sep 09 14:38:17.924089 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:17.924051 2578 generic.go:356] "Generic (PLEG): container finished" podID="14079d71-4d00-41e3-b4f0-efab1cf10451" containerID="2446466e099b2c1a8d56890cb6d3ad1989978bd14dde3c8c71ca1202621ddadd" exitCode=0 Sep 09 14:38:17.924567 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:17.924144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2dlbn" event={"ID":"14079d71-4d00-41e3-b4f0-efab1cf10451","Type":"ContainerDied","Data":"2446466e099b2c1a8d56890cb6d3ad1989978bd14dde3c8c71ca1202621ddadd"} Sep 09 14:38:18.883055 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:18.883024 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9v8sj" Sep 09 14:38:18.933209 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:18.933178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2dlbn" event={"ID":"14079d71-4d00-41e3-b4f0-efab1cf10451","Type":"ContainerStarted","Data":"419553dc96253f6cb323f8fe79674fb5cf9fa48a0bd015a645cd5fc14856ce74"} Sep 09 14:38:18.933596 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:18.933231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2dlbn" event={"ID":"14079d71-4d00-41e3-b4f0-efab1cf10451","Type":"ContainerStarted","Data":"10e84a2cf66a9f33b06c916cadf165ead9c503e56fb7b555e95e292d3fd1c57c"} Sep 09 14:38:18.962045 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:18.961999 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2dlbn" podStartSLOduration=4.015263647 podStartE2EDuration="4.961982481s" totalImagesPullingTime="885.00716ms" totalInitContainerRuntime="61.711674ms" isStatefulPod=true podCreationTimestamp="2026-09-09 14:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:18.960630972 +0000 UTC m=+298.627437018" watchObservedRunningTime="2026-09-09 14:38:18.961982481 +0000 UTC m=+298.628788527" Sep 09 14:38:20.738220 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:20.738187 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:38:20.738710 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:20.738425 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:38:20.754126 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:20.754104 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 14:38:22.899634 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:22.899594 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:22.902094 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:22.899657 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:22.904656 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:22.904633 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:22.949552 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:22.949526 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:38:23.349958 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.349925 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f897565d9-xd62d"] Sep 09 14:38:23.353090 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.353072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.360053 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.360037 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 14:38:23.363251 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.363228 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f897565d9-xd62d"] Sep 09 14:38:23.466255 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.466222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-service-ca\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.466436 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.466282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-oauth-serving-cert\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.466436 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.466341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-console-config\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.466436 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.466390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q54fd\" (UniqueName: \"kubernetes.io/projected/729d23a3-1293-4800-bffc-f2632f744ad8-kube-api-access-q54fd\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.466542 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.466454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-oauth-config\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.466542 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.466492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-serving-cert\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.466542 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.466509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-trusted-ca-bundle\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.567869 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.567833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-service-ca\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.568043 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.567936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-oauth-serving-cert\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.568043 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.567969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-console-config\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.568043 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.567994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q54fd\" (UniqueName: \"kubernetes.io/projected/729d23a3-1293-4800-bffc-f2632f744ad8-kube-api-access-q54fd\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.568204 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.568047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-oauth-config\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.568204 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.568119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-serving-cert\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.568204 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.568151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-trusted-ca-bundle\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.568798 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.568639 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-service-ca\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.569455 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.569405 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-oauth-serving-cert\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.569849 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.569822 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-trusted-ca-bundle\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.570096 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.570069 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-console-config\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.571230 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.571203 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-oauth-config\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.571317 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.571274 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-serving-cert\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.579929 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.579909 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q54fd\" (UniqueName: \"kubernetes.io/projected/729d23a3-1293-4800-bffc-f2632f744ad8-kube-api-access-q54fd\") pod \"console-7f897565d9-xd62d\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.662297 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.662203 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:23.936956 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.936927 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f897565d9-xd62d"] Sep 09 14:38:23.939288 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:23.939258 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729d23a3_1293_4800_bffc_f2632f744ad8.slice/crio-9c1f0ec6c87fb92c1da4a8c53792a0c2c1bed326c1ba9242cb86eff6a8d0776a WatchSource:0}: Error finding container 9c1f0ec6c87fb92c1da4a8c53792a0c2c1bed326c1ba9242cb86eff6a8d0776a: Status 404 returned error can't find the container with id 9c1f0ec6c87fb92c1da4a8c53792a0c2c1bed326c1ba9242cb86eff6a8d0776a Sep 09 14:38:23.951002 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:23.950976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f897565d9-xd62d" event={"ID":"729d23a3-1293-4800-bffc-f2632f744ad8","Type":"ContainerStarted","Data":"9c1f0ec6c87fb92c1da4a8c53792a0c2c1bed326c1ba9242cb86eff6a8d0776a"} Sep 09 14:38:24.956178 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:24.956149 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f897565d9-xd62d" event={"ID":"729d23a3-1293-4800-bffc-f2632f744ad8","Type":"ContainerStarted","Data":"b0be5248f9d89daaa746a5819b1fa0cd049d22adb5d50dcafe5db8fd1fd4c8ff"} Sep 09 14:38:24.976141 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:24.976098 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f897565d9-xd62d" podStartSLOduration=1.9760850140000001 podStartE2EDuration="1.976085014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:24.975285261 +0000 UTC m=+304.642091307" watchObservedRunningTime="2026-09-09 14:38:24.976085014 +0000 UTC m=+304.642891059" Sep 09 14:38:28.849487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:28.849450 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f897565d9-xd62d"] Sep 09 14:38:28.919728 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:28.919691 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b4b89cbc-6d45z"] Sep 09 14:38:28.925314 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:28.925291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:28.938634 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:28.938609 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b4b89cbc-6d45z"] Sep 09 14:38:29.019165 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.019123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-config\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.019325 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.019178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-serving-cert\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.019325 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.019204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-oauth-serving-cert\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.019325 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.019233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-oauth-config\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.019515 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.019319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6b6v\" (UniqueName: \"kubernetes.io/projected/67537fb3-291d-4e0e-a6ea-61c23010d7ab-kube-api-access-z6b6v\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.019515 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.019399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-service-ca\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.019515 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.019444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-trusted-ca-bundle\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.120761 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.120663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-config\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.120761 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.120726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-serving-cert\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.120761 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.120748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-oauth-serving-cert\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.121022 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.120773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-oauth-config\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.121022 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.120829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6b6v\" (UniqueName: \"kubernetes.io/projected/67537fb3-291d-4e0e-a6ea-61c23010d7ab-kube-api-access-z6b6v\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.121022 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.120863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-service-ca\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.121022 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.120900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-trusted-ca-bundle\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.121916 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.121886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-oauth-serving-cert\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.122088 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.121901 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-config\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.122194 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.121911 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-trusted-ca-bundle\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.122361 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.121967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-service-ca\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.123978 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.123953 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-serving-cert\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.124305 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.124285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-oauth-config\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.136700 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.136678 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6b6v\" (UniqueName: \"kubernetes.io/projected/67537fb3-291d-4e0e-a6ea-61c23010d7ab-kube-api-access-z6b6v\") pod \"console-5b4b89cbc-6d45z\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.234550 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.234517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:29.519289 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.519253 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b4b89cbc-6d45z"] Sep 09 14:38:29.522286 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:38:29.522258 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67537fb3_291d_4e0e_a6ea_61c23010d7ab.slice/crio-377555483c1eecb45c960298d56eabf8afd62c26a0ed5e6916b2e92e9f1564bf WatchSource:0}: Error finding container 377555483c1eecb45c960298d56eabf8afd62c26a0ed5e6916b2e92e9f1564bf: Status 404 returned error can't find the container with id 377555483c1eecb45c960298d56eabf8afd62c26a0ed5e6916b2e92e9f1564bf Sep 09 14:38:29.976550 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.976521 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:38:29.978053 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.978027 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4b89cbc-6d45z" event={"ID":"67537fb3-291d-4e0e-a6ea-61c23010d7ab","Type":"ContainerStarted","Data":"a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c"} Sep 09 14:38:29.978169 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.978066 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4b89cbc-6d45z" event={"ID":"67537fb3-291d-4e0e-a6ea-61c23010d7ab","Type":"ContainerStarted","Data":"377555483c1eecb45c960298d56eabf8afd62c26a0ed5e6916b2e92e9f1564bf"} Sep 09 14:38:29.997006 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:29.996963 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b4b89cbc-6d45z" podStartSLOduration=1.9969481930000001 podStartE2EDuration="1.996948193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:38:29.995823985 +0000 UTC m=+309.662630035" watchObservedRunningTime="2026-09-09 14:38:29.996948193 +0000 UTC m=+309.663754239" Sep 09 14:38:33.663101 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:33.663066 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:39.235238 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:39.235208 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:39.235654 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:39.235506 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:39.240383 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:39.240363 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:40.015939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:40.015911 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:38:40.058661 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:40.058628 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb66dbb94-9gzlf"] Sep 09 14:38:53.870194 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:53.870145 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f897565d9-xd62d" podUID="729d23a3-1293-4800-bffc-f2632f744ad8" containerName="console" containerID="cri-o://b0be5248f9d89daaa746a5819b1fa0cd049d22adb5d50dcafe5db8fd1fd4c8ff" gracePeriod=15 Sep 09 14:38:54.062289 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.062262 2578 generic.go:356] "Generic (PLEG): container finished" podID="729d23a3-1293-4800-bffc-f2632f744ad8" containerID="b0be5248f9d89daaa746a5819b1fa0cd049d22adb5d50dcafe5db8fd1fd4c8ff" exitCode=0 Sep 09 14:38:54.062432 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.062337 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f897565d9-xd62d" event={"ID":"729d23a3-1293-4800-bffc-f2632f744ad8","Type":"ContainerDied","Data":"b0be5248f9d89daaa746a5819b1fa0cd049d22adb5d50dcafe5db8fd1fd4c8ff"} Sep 09 14:38:54.111805 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.111783 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:54.130459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130393 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-serving-cert\") pod \"729d23a3-1293-4800-bffc-f2632f744ad8\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " Sep 09 14:38:54.130459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130447 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-service-ca\") pod \"729d23a3-1293-4800-bffc-f2632f744ad8\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " Sep 09 14:38:54.130630 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130470 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-trusted-ca-bundle\") pod \"729d23a3-1293-4800-bffc-f2632f744ad8\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " Sep 09 14:38:54.130630 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130492 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-console-config\") pod \"729d23a3-1293-4800-bffc-f2632f744ad8\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " Sep 09 14:38:54.130630 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130523 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-oauth-serving-cert\") pod \"729d23a3-1293-4800-bffc-f2632f744ad8\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " Sep 09 14:38:54.130784 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130683 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q54fd\" (UniqueName: \"kubernetes.io/projected/729d23a3-1293-4800-bffc-f2632f744ad8-kube-api-access-q54fd\") pod \"729d23a3-1293-4800-bffc-f2632f744ad8\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " Sep 09 14:38:54.130784 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130719 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-oauth-config\") pod \"729d23a3-1293-4800-bffc-f2632f744ad8\" (UID: \"729d23a3-1293-4800-bffc-f2632f744ad8\") " Sep 09 14:38:54.130923 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130902 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "729d23a3-1293-4800-bffc-f2632f744ad8" (UID: "729d23a3-1293-4800-bffc-f2632f744ad8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:54.130978 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130914 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-console-config" (OuterVolumeSpecName: "console-config") pod "729d23a3-1293-4800-bffc-f2632f744ad8" (UID: "729d23a3-1293-4800-bffc-f2632f744ad8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:54.130978 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130928 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-service-ca" (OuterVolumeSpecName: "service-ca") pod "729d23a3-1293-4800-bffc-f2632f744ad8" (UID: "729d23a3-1293-4800-bffc-f2632f744ad8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:54.130978 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.130936 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "729d23a3-1293-4800-bffc-f2632f744ad8" (UID: "729d23a3-1293-4800-bffc-f2632f744ad8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:38:54.131100 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.131047 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-oauth-serving-cert\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:38:54.131100 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.131067 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-service-ca\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:38:54.131100 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.131082 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-trusted-ca-bundle\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:38:54.131100 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.131097 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/729d23a3-1293-4800-bffc-f2632f744ad8-console-config\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:38:54.133050 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.133027 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "729d23a3-1293-4800-bffc-f2632f744ad8" (UID: "729d23a3-1293-4800-bffc-f2632f744ad8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:38:54.133193 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.133165 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729d23a3-1293-4800-bffc-f2632f744ad8-kube-api-access-q54fd" (OuterVolumeSpecName: "kube-api-access-q54fd") pod "729d23a3-1293-4800-bffc-f2632f744ad8" (UID: "729d23a3-1293-4800-bffc-f2632f744ad8"). InnerVolumeSpecName "kube-api-access-q54fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:38:54.133285 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.133234 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "729d23a3-1293-4800-bffc-f2632f744ad8" (UID: "729d23a3-1293-4800-bffc-f2632f744ad8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:38:54.232120 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.232088 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-serving-cert\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:38:54.232120 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.232116 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q54fd\" (UniqueName: \"kubernetes.io/projected/729d23a3-1293-4800-bffc-f2632f744ad8-kube-api-access-q54fd\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:38:54.232294 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:54.232130 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/729d23a3-1293-4800-bffc-f2632f744ad8-console-oauth-config\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:38:55.067161 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:55.067129 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f897565d9-xd62d" event={"ID":"729d23a3-1293-4800-bffc-f2632f744ad8","Type":"ContainerDied","Data":"9c1f0ec6c87fb92c1da4a8c53792a0c2c1bed326c1ba9242cb86eff6a8d0776a"} Sep 09 14:38:55.067161 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:55.067148 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f897565d9-xd62d" Sep 09 14:38:55.067622 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:55.067191 2578 scope.go:117] "RemoveContainer" containerID="b0be5248f9d89daaa746a5819b1fa0cd049d22adb5d50dcafe5db8fd1fd4c8ff" Sep 09 14:38:55.088644 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:55.088616 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f897565d9-xd62d"] Sep 09 14:38:55.092986 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:55.092966 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f897565d9-xd62d"] Sep 09 14:38:56.981657 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:38:56.981624 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="729d23a3-1293-4800-bffc-f2632f744ad8" path="/var/lib/kubelet/pods/729d23a3-1293-4800-bffc-f2632f744ad8/volumes" Sep 09 14:39:01.090795 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:01.090754 2578 generic.go:356] "Generic (PLEG): container finished" podID="ba32ff86-2186-4f15-84f4-8b2c63867723" containerID="537fd0646f65e0da1bd9a55ba2dba3eafc992b52de7f7e2e2e165dee10b94620" exitCode=0 Sep 09 14:39:01.091269 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:01.090807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" event={"ID":"ba32ff86-2186-4f15-84f4-8b2c63867723","Type":"ContainerDied","Data":"537fd0646f65e0da1bd9a55ba2dba3eafc992b52de7f7e2e2e165dee10b94620"} Sep 09 14:39:01.091269 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:01.091152 2578 scope.go:117] "RemoveContainer" containerID="537fd0646f65e0da1bd9a55ba2dba3eafc992b52de7f7e2e2e165dee10b94620" Sep 09 14:39:02.096239 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:02.096203 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-2mw8h" event={"ID":"ba32ff86-2186-4f15-84f4-8b2c63867723","Type":"ContainerStarted","Data":"c6e4f07a1ebf6ddd5fd413e057e59aa6e6f94a44c32c965597ec112b4b2ccc30"} Sep 09 14:39:05.080465 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.080426 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bb66dbb94-9gzlf" podUID="c4dda96e-8c13-4b84-9c75-c136133dc0bc" containerName="console" containerID="cri-o://4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2" gracePeriod=15 Sep 09 14:39:05.319583 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.319562 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:39:05.420394 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.420286 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-oauth-serving-cert\") pod \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " Sep 09 14:39:05.420394 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.420334 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-oauth-config\") pod \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " Sep 09 14:39:05.420394 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.420390 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-serving-cert\") pod \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " Sep 09 14:39:05.420661 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.420422 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-config\") pod \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " Sep 09 14:39:05.420661 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.420447 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-service-ca\") pod \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " Sep 09 14:39:05.420661 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.420468 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqb68\" (UniqueName: \"kubernetes.io/projected/c4dda96e-8c13-4b84-9c75-c136133dc0bc-kube-api-access-sqb68\") pod \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\" (UID: \"c4dda96e-8c13-4b84-9c75-c136133dc0bc\") " Sep 09 14:39:05.420811 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.420759 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c4dda96e-8c13-4b84-9c75-c136133dc0bc" (UID: "c4dda96e-8c13-4b84-9c75-c136133dc0bc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:05.420869 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.420835 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-service-ca" (OuterVolumeSpecName: "service-ca") pod "c4dda96e-8c13-4b84-9c75-c136133dc0bc" (UID: "c4dda96e-8c13-4b84-9c75-c136133dc0bc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:05.420869 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.420845 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-config" (OuterVolumeSpecName: "console-config") pod "c4dda96e-8c13-4b84-9c75-c136133dc0bc" (UID: "c4dda96e-8c13-4b84-9c75-c136133dc0bc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:39:05.422696 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.422672 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c4dda96e-8c13-4b84-9c75-c136133dc0bc" (UID: "c4dda96e-8c13-4b84-9c75-c136133dc0bc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:05.422800 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.422777 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4dda96e-8c13-4b84-9c75-c136133dc0bc-kube-api-access-sqb68" (OuterVolumeSpecName: "kube-api-access-sqb68") pod "c4dda96e-8c13-4b84-9c75-c136133dc0bc" (UID: "c4dda96e-8c13-4b84-9c75-c136133dc0bc"). InnerVolumeSpecName "kube-api-access-sqb68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:39:05.422837 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.422813 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c4dda96e-8c13-4b84-9c75-c136133dc0bc" (UID: "c4dda96e-8c13-4b84-9c75-c136133dc0bc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:39:05.525538 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.521796 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-oauth-serving-cert\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:39:05.525538 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.521844 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-oauth-config\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:39:05.525538 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.521859 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-serving-cert\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:39:05.525538 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.521879 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-console-config\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:39:05.525538 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.521894 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4dda96e-8c13-4b84-9c75-c136133dc0bc-service-ca\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:39:05.525538 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:05.521912 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqb68\" (UniqueName: \"kubernetes.io/projected/c4dda96e-8c13-4b84-9c75-c136133dc0bc-kube-api-access-sqb68\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:39:06.111154 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.111111 2578 generic.go:356] "Generic (PLEG): container finished" podID="c4dda96e-8c13-4b84-9c75-c136133dc0bc" containerID="4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2" exitCode=0 Sep 09 14:39:06.111557 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.111175 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb66dbb94-9gzlf" Sep 09 14:39:06.111557 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.111201 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb66dbb94-9gzlf" event={"ID":"c4dda96e-8c13-4b84-9c75-c136133dc0bc","Type":"ContainerDied","Data":"4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2"} Sep 09 14:39:06.111557 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.111256 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb66dbb94-9gzlf" event={"ID":"c4dda96e-8c13-4b84-9c75-c136133dc0bc","Type":"ContainerDied","Data":"b1f4848d0abe2455ccdba218ed50bcd6c3213c1868ac5fbd278a9684301f7db2"} Sep 09 14:39:06.111557 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.111281 2578 scope.go:117] "RemoveContainer" containerID="4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2" Sep 09 14:39:06.119462 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.119446 2578 scope.go:117] "RemoveContainer" containerID="4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2" Sep 09 14:39:06.119730 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:39:06.119711 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2\": container with ID starting with 4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2 not found: ID does not exist" containerID="4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2" Sep 09 14:39:06.119797 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.119740 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2"} err="failed to get container status \"4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2\": rpc error: code = NotFound desc = could not find container \"4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2\": container with ID starting with 4968890135e930d70bb18071a3954b6d776f927e7d09f72c70ba507e120b8fc2 not found: ID does not exist" Sep 09 14:39:06.131055 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.131034 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb66dbb94-9gzlf"] Sep 09 14:39:06.135077 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.135056 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bb66dbb94-9gzlf"] Sep 09 14:39:06.980760 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:06.980726 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4dda96e-8c13-4b84-9c75-c136133dc0bc" path="/var/lib/kubelet/pods/c4dda96e-8c13-4b84-9c75-c136133dc0bc/volumes" Sep 09 14:39:32.708169 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.708135 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84cdc4676-2xnqt"] Sep 09 14:39:32.708684 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.708586 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4dda96e-8c13-4b84-9c75-c136133dc0bc" containerName="console" Sep 09 14:39:32.708684 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.708604 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4dda96e-8c13-4b84-9c75-c136133dc0bc" containerName="console" Sep 09 14:39:32.708684 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.708615 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729d23a3-1293-4800-bffc-f2632f744ad8" containerName="console" Sep 09 14:39:32.708684 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.708623 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="729d23a3-1293-4800-bffc-f2632f744ad8" containerName="console" Sep 09 14:39:32.708948 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.708691 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="729d23a3-1293-4800-bffc-f2632f744ad8" containerName="console" Sep 09 14:39:32.708948 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.708702 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4dda96e-8c13-4b84-9c75-c136133dc0bc" containerName="console" Sep 09 14:39:32.713024 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.713004 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.736133 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.736109 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cdc4676-2xnqt"] Sep 09 14:39:32.750290 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.750267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-config\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.750400 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.750298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7v7h\" (UniqueName: \"kubernetes.io/projected/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-kube-api-access-v7v7h\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.750400 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.750333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-trusted-ca-bundle\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.750487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.750428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-serving-cert\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.750487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.750444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-oauth-serving-cert\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.750487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.750460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-oauth-config\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.750487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.750475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-service-ca\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.851178 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.851153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-trusted-ca-bundle\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.851328 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.851219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-serving-cert\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.851328 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.851245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-oauth-serving-cert\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.851328 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.851267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-oauth-config\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.851521 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.851388 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-service-ca\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.851521 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.851425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-config\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.851521 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.851453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7v7h\" (UniqueName: \"kubernetes.io/projected/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-kube-api-access-v7v7h\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.852003 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.851981 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-oauth-serving-cert\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.852148 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.852124 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-trusted-ca-bundle\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.852216 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.852196 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-config\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.852271 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.852255 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-service-ca\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.854289 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.854268 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-serving-cert\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.854289 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.854278 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-oauth-config\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:32.860923 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:32.860901 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7v7h\" (UniqueName: \"kubernetes.io/projected/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-kube-api-access-v7v7h\") pod \"console-84cdc4676-2xnqt\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:33.021665 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:33.021632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:33.292956 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:33.292931 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cdc4676-2xnqt"] Sep 09 14:39:33.294869 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:39:33.294843 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdcf315_f0cd_4379_b8bb_a47b5fd9249a.slice/crio-e6b78d195a9a7d978cd1f1468525f247a121a89727c376eb4c5027298e30ba0d WatchSource:0}: Error finding container e6b78d195a9a7d978cd1f1468525f247a121a89727c376eb4c5027298e30ba0d: Status 404 returned error can't find the container with id e6b78d195a9a7d978cd1f1468525f247a121a89727c376eb4c5027298e30ba0d Sep 09 14:39:34.212119 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:34.212086 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cdc4676-2xnqt" event={"ID":"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a","Type":"ContainerStarted","Data":"c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e"} Sep 09 14:39:34.212501 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:34.212140 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cdc4676-2xnqt" event={"ID":"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a","Type":"ContainerStarted","Data":"e6b78d195a9a7d978cd1f1468525f247a121a89727c376eb4c5027298e30ba0d"} Sep 09 14:39:34.230045 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:34.229984 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84cdc4676-2xnqt" podStartSLOduration=2.229965713 podStartE2EDuration="2.229965713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:39:34.228573655 +0000 UTC m=+373.895379701" watchObservedRunningTime="2026-09-09 14:39:34.229965713 +0000 UTC m=+373.896771758" Sep 09 14:39:34.666096 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:34.666052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:39:34.668403 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:34.668382 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c566cce4-7251-42bf-8696-f2505562d073-metrics-certs\") pod \"network-metrics-daemon-fs9bf\" (UID: \"c566cce4-7251-42bf-8696-f2505562d073\") " pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:39:34.780460 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:34.780428 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bjh9b\"" Sep 09 14:39:34.788606 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:34.788585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs9bf" Sep 09 14:39:35.085947 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:35.085924 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fs9bf"] Sep 09 14:39:35.087800 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:39:35.087763 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc566cce4_7251_42bf_8696_f2505562d073.slice/crio-539bc8ad89885a08ee538f97e0f7702f01e2a9a068f520996e640fb124491318 WatchSource:0}: Error finding container 539bc8ad89885a08ee538f97e0f7702f01e2a9a068f520996e640fb124491318: Status 404 returned error can't find the container with id 539bc8ad89885a08ee538f97e0f7702f01e2a9a068f520996e640fb124491318 Sep 09 14:39:35.089669 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:35.089652 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 14:39:35.217178 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:35.217144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fs9bf" event={"ID":"c566cce4-7251-42bf-8696-f2505562d073","Type":"ContainerStarted","Data":"539bc8ad89885a08ee538f97e0f7702f01e2a9a068f520996e640fb124491318"} Sep 09 14:39:36.222925 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:36.222897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fs9bf" event={"ID":"c566cce4-7251-42bf-8696-f2505562d073","Type":"ContainerStarted","Data":"c2438da02183ef0ea50ed11237ca75ae420ff08077dfdd7396c9015a2d59d544"} Sep 09 14:39:36.223304 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:36.222947 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fs9bf" event={"ID":"c566cce4-7251-42bf-8696-f2505562d073","Type":"ContainerStarted","Data":"43dc333b179e2c16ddfb0b6345b5bf6d81a8df81a97e552b4e6012278848726c"} Sep 09 14:39:36.242093 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:36.242047 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fs9bf" podStartSLOduration=373.283937196 podStartE2EDuration="6m14.242029334s" totalImagesPullingTime="958.092138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:39:36.242028632 +0000 UTC m=+375.908834691" watchObservedRunningTime="2026-09-09 14:39:36.242029334 +0000 UTC m=+375.908835380" Sep 09 14:39:43.022678 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:43.022635 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:43.023031 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:43.022703 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:43.027165 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:43.027143 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:43.255314 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:43.255284 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:39:43.342169 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:39:43.342095 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b4b89cbc-6d45z"] Sep 09 14:40:08.364632 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.364569 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b4b89cbc-6d45z" podUID="67537fb3-291d-4e0e-a6ea-61c23010d7ab" containerName="console" containerID="cri-o://a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c" gracePeriod=15 Sep 09 14:40:08.611590 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.611567 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:40:08.732228 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732191 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-oauth-serving-cert\") pod \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " Sep 09 14:40:08.732228 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732234 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-service-ca\") pod \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " Sep 09 14:40:08.732468 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732272 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-serving-cert\") pod \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " Sep 09 14:40:08.732468 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732292 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6b6v\" (UniqueName: \"kubernetes.io/projected/67537fb3-291d-4e0e-a6ea-61c23010d7ab-kube-api-access-z6b6v\") pod \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " Sep 09 14:40:08.732468 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732322 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-config\") pod \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " Sep 09 14:40:08.732468 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732337 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-oauth-config\") pod \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " Sep 09 14:40:08.732468 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732390 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-trusted-ca-bundle\") pod \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\" (UID: \"67537fb3-291d-4e0e-a6ea-61c23010d7ab\") " Sep 09 14:40:08.732751 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732719 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "67537fb3-291d-4e0e-a6ea-61c23010d7ab" (UID: "67537fb3-291d-4e0e-a6ea-61c23010d7ab"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:40:08.732751 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732737 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-config" (OuterVolumeSpecName: "console-config") pod "67537fb3-291d-4e0e-a6ea-61c23010d7ab" (UID: "67537fb3-291d-4e0e-a6ea-61c23010d7ab"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:40:08.732907 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732770 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-service-ca" (OuterVolumeSpecName: "service-ca") pod "67537fb3-291d-4e0e-a6ea-61c23010d7ab" (UID: "67537fb3-291d-4e0e-a6ea-61c23010d7ab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:40:08.732907 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.732858 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "67537fb3-291d-4e0e-a6ea-61c23010d7ab" (UID: "67537fb3-291d-4e0e-a6ea-61c23010d7ab"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:40:08.734554 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.734530 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "67537fb3-291d-4e0e-a6ea-61c23010d7ab" (UID: "67537fb3-291d-4e0e-a6ea-61c23010d7ab"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:40:08.734554 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.734540 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67537fb3-291d-4e0e-a6ea-61c23010d7ab-kube-api-access-z6b6v" (OuterVolumeSpecName: "kube-api-access-z6b6v") pod "67537fb3-291d-4e0e-a6ea-61c23010d7ab" (UID: "67537fb3-291d-4e0e-a6ea-61c23010d7ab"). InnerVolumeSpecName "kube-api-access-z6b6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:40:08.734692 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.734641 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "67537fb3-291d-4e0e-a6ea-61c23010d7ab" (UID: "67537fb3-291d-4e0e-a6ea-61c23010d7ab"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:40:08.832859 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.832830 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-serving-cert\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:40:08.832859 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.832854 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6b6v\" (UniqueName: \"kubernetes.io/projected/67537fb3-291d-4e0e-a6ea-61c23010d7ab-kube-api-access-z6b6v\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:40:08.832859 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.832864 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-config\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:40:08.833043 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.832872 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67537fb3-291d-4e0e-a6ea-61c23010d7ab-console-oauth-config\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:40:08.833043 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.832881 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-trusted-ca-bundle\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:40:08.833043 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.832889 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-oauth-serving-cert\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:40:08.833043 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:08.832897 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67537fb3-291d-4e0e-a6ea-61c23010d7ab-service-ca\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:40:09.346916 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:09.346834 2578 generic.go:356] "Generic (PLEG): container finished" podID="67537fb3-291d-4e0e-a6ea-61c23010d7ab" containerID="a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c" exitCode=0 Sep 09 14:40:09.346916 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:09.346880 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4b89cbc-6d45z" Sep 09 14:40:09.347141 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:09.346923 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4b89cbc-6d45z" event={"ID":"67537fb3-291d-4e0e-a6ea-61c23010d7ab","Type":"ContainerDied","Data":"a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c"} Sep 09 14:40:09.347141 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:09.346976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4b89cbc-6d45z" event={"ID":"67537fb3-291d-4e0e-a6ea-61c23010d7ab","Type":"ContainerDied","Data":"377555483c1eecb45c960298d56eabf8afd62c26a0ed5e6916b2e92e9f1564bf"} Sep 09 14:40:09.347141 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:09.347002 2578 scope.go:117] "RemoveContainer" containerID="a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c" Sep 09 14:40:09.354644 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:09.354630 2578 scope.go:117] "RemoveContainer" containerID="a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c" Sep 09 14:40:09.354951 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:40:09.354934 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c\": container with ID starting with a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c not found: ID does not exist" containerID="a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c" Sep 09 14:40:09.354996 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:09.354959 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c"} err="failed to get container status \"a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c\": rpc error: code = NotFound desc = could not find container \"a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c\": container with ID starting with a4c8ea402b961f2c1c179d6c9158df7bd7387b81c4b2e36126bd37343ed00a8c not found: ID does not exist" Sep 09 14:40:09.363192 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:09.363172 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b4b89cbc-6d45z"] Sep 09 14:40:09.366225 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:09.366208 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b4b89cbc-6d45z"] Sep 09 14:40:10.981813 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:40:10.981778 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67537fb3-291d-4e0e-a6ea-61c23010d7ab" path="/var/lib/kubelet/pods/67537fb3-291d-4e0e-a6ea-61c23010d7ab/volumes" Sep 09 14:41:55.880685 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:55.880635 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc"] Sep 09 14:41:55.881244 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:55.881219 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67537fb3-291d-4e0e-a6ea-61c23010d7ab" containerName="console" Sep 09 14:41:55.881430 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:55.881415 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67537fb3-291d-4e0e-a6ea-61c23010d7ab" containerName="console" Sep 09 14:41:55.881673 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:55.881660 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67537fb3-291d-4e0e-a6ea-61c23010d7ab" containerName="console" Sep 09 14:41:55.885053 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:55.885028 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" Sep 09 14:41:55.888529 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:55.888509 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-l6vvr\"" Sep 09 14:41:55.889178 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:55.889158 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 09 14:41:55.889398 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:55.889279 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 09 14:41:55.902731 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:55.902702 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc"] Sep 09 14:41:56.021980 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:56.021939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvv96\" (UniqueName: \"kubernetes.io/projected/8140b4f0-6fff-4f41-acb4-4cd03c913cdf-kube-api-access-xvv96\") pod \"feast-operator-controller-manager-9c5b6dd6c-z4wcc\" (UID: \"8140b4f0-6fff-4f41-acb4-4cd03c913cdf\") " pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" Sep 09 14:41:56.123246 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:56.123192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvv96\" (UniqueName: \"kubernetes.io/projected/8140b4f0-6fff-4f41-acb4-4cd03c913cdf-kube-api-access-xvv96\") pod \"feast-operator-controller-manager-9c5b6dd6c-z4wcc\" (UID: \"8140b4f0-6fff-4f41-acb4-4cd03c913cdf\") " pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" Sep 09 14:41:56.134440 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:56.134388 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvv96\" (UniqueName: \"kubernetes.io/projected/8140b4f0-6fff-4f41-acb4-4cd03c913cdf-kube-api-access-xvv96\") pod \"feast-operator-controller-manager-9c5b6dd6c-z4wcc\" (UID: \"8140b4f0-6fff-4f41-acb4-4cd03c913cdf\") " pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" Sep 09 14:41:56.195488 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:56.195458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" Sep 09 14:41:56.472544 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:56.472517 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc"] Sep 09 14:41:56.474880 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:41:56.474847 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8140b4f0_6fff_4f41_acb4_4cd03c913cdf.slice/crio-6958db78375510e61cae23fe8311aa141005da3e2af939f446ec722db0b88d05 WatchSource:0}: Error finding container 6958db78375510e61cae23fe8311aa141005da3e2af939f446ec722db0b88d05: Status 404 returned error can't find the container with id 6958db78375510e61cae23fe8311aa141005da3e2af939f446ec722db0b88d05 Sep 09 14:41:56.726031 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:56.725944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" event={"ID":"8140b4f0-6fff-4f41-acb4-4cd03c913cdf","Type":"ContainerStarted","Data":"6958db78375510e61cae23fe8311aa141005da3e2af939f446ec722db0b88d05"} Sep 09 14:41:59.739540 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:59.739500 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" event={"ID":"8140b4f0-6fff-4f41-acb4-4cd03c913cdf","Type":"ContainerStarted","Data":"2f7a156a77a3ef14c5097a135e54d7ea2872bae7c721688ff546901e3983cc03"} Sep 09 14:41:59.740021 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:59.739620 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" Sep 09 14:41:59.757204 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:41:59.757158 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" podStartSLOduration=2.298527674 podStartE2EDuration="4.757141177s" totalImagesPullingTime="2.458613503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:41:59.755278239 +0000 UTC m=+519.422084279" watchObservedRunningTime="2026-09-09 14:41:59.757141177 +0000 UTC m=+519.423947225" Sep 09 14:42:10.746515 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:10.746478 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" Sep 09 14:42:34.222141 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.222107 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h"] Sep 09 14:42:34.228462 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.228444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.230833 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.230805 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 14:42:34.230959 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.230849 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 09 14:42:34.230959 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.230894 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 14:42:34.230959 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.230896 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-8qtmg\"" Sep 09 14:42:34.230959 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.230946 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 14:42:34.231225 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.230972 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 14:42:34.231225 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.231013 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 14:42:34.231225 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.231147 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 14:42:34.237531 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.237509 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h"] Sep 09 14:42:34.301965 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.301931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-ui-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.302148 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.301984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f912c6-51fc-4d74-a347-01c8ca464ce5-feast-data\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.302148 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.302005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6g2\" (UniqueName: \"kubernetes.io/projected/a8f912c6-51fc-4d74-a347-01c8ca464ce5-kube-api-access-xw6g2\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.302148 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.302026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-offline-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.302148 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.302085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-registry-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.302148 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.302118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-online-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.403095 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.403055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-ui-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.403269 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.403119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f912c6-51fc-4d74-a347-01c8ca464ce5-feast-data\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.403269 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.403148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6g2\" (UniqueName: \"kubernetes.io/projected/a8f912c6-51fc-4d74-a347-01c8ca464ce5-kube-api-access-xw6g2\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.403269 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.403175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-offline-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.403269 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.403206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-registry-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.403269 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.403233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-online-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.403667 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.403637 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f912c6-51fc-4d74-a347-01c8ca464ce5-feast-data\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.405691 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.405670 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-online-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.405835 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.405750 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-registry-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.406057 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.406042 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-offline-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.406103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.406053 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-ui-tls\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.411482 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.411462 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6g2\" (UniqueName: \"kubernetes.io/projected/a8f912c6-51fc-4d74-a347-01c8ca464ce5-kube-api-access-xw6g2\") pod \"feast-simple-feast-setup-58fc98d76c-7bp5h\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.537899 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.537798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:42:34.816429 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.816342 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h"] Sep 09 14:42:34.819981 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:42:34.819955 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8f912c6_51fc_4d74_a347_01c8ca464ce5.slice/crio-698d18a4d4b8b4f28f085e2a0278171c4bc5de2405917b16146f2d7a1d758aaa WatchSource:0}: Error finding container 698d18a4d4b8b4f28f085e2a0278171c4bc5de2405917b16146f2d7a1d758aaa: Status 404 returned error can't find the container with id 698d18a4d4b8b4f28f085e2a0278171c4bc5de2405917b16146f2d7a1d758aaa Sep 09 14:42:34.868361 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:42:34.868314 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerStarted","Data":"698d18a4d4b8b4f28f085e2a0278171c4bc5de2405917b16146f2d7a1d758aaa"} Sep 09 14:43:01.973992 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:01.973957 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerStarted","Data":"7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8"} Sep 09 14:43:06.992218 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:06.992184 2578 generic.go:356] "Generic (PLEG): container finished" podID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerID="7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8" exitCode=0 Sep 09 14:43:06.992657 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:06.992231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerDied","Data":"7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8"} Sep 09 14:43:07.998481 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:07.998435 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerStarted","Data":"a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d"} Sep 09 14:43:13.019419 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:13.019334 2578 generic.go:356] "Generic (PLEG): container finished" podID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerID="a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d" exitCode=0 Sep 09 14:43:13.019889 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:13.019448 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerDied","Data":"a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d"} Sep 09 14:43:14.029576 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:14.029535 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerStarted","Data":"0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f"} Sep 09 14:43:14.030039 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:14.029602 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerStarted","Data":"c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50"} Sep 09 14:43:14.030039 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:14.029624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerStarted","Data":"ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206"} Sep 09 14:43:14.030039 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:14.029647 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerStarted","Data":"bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc"} Sep 09 14:43:14.075363 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:14.075275 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podStartSLOduration=13.003101622 podStartE2EDuration="40.075252891s" totalImagesPullingTime="27.072151269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:43:14.068642623 +0000 UTC m=+593.735448682" watchObservedRunningTime="2026-09-09 14:43:14.075252891 +0000 UTC m=+593.742058939" Sep 09 14:43:14.538500 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:14.538457 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:14.538709 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:14.538531 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:14.538709 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:14.538551 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:14.538709 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:14.538569 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:16.538695 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:16.538655 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:16.539170 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:16.538912 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:16.539170 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:16.538944 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:16.539170 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:16.538963 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:16.540897 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:16.540862 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 14:43:16.541019 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:16.540892 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 14:43:16.541019 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:16.540945 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Sep 09 14:43:16.541133 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:16.541034 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Sep 09 14:43:19.539416 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:19.539231 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Sep 09 14:43:19.539416 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:19.539258 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 14:43:19.539416 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:19.539257 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Sep 09 14:43:19.539416 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:19.539316 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 14:43:20.770258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:20.770229 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:43:20.771062 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:20.771038 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:43:22.539688 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:22.539641 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:22.540223 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:22.540186 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:22.540411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:22.540371 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:22.540507 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:22.540481 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:22.545278 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:22.545255 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:23.074309 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:23.074280 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:23.080105 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:23.074338 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:23.080105 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:23.076891 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:43:45.819278 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:43:45.819197 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84cdc4676-2xnqt"] Sep 09 14:44:10.840909 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:10.840864 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84cdc4676-2xnqt" podUID="9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" containerName="console" containerID="cri-o://c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e" gracePeriod=15 Sep 09 14:44:11.077932 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.077907 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:44:11.184144 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184065 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7v7h\" (UniqueName: \"kubernetes.io/projected/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-kube-api-access-v7v7h\") pod \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " Sep 09 14:44:11.184144 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184105 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-trusted-ca-bundle\") pod \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " Sep 09 14:44:11.184144 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184145 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-config\") pod \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " Sep 09 14:44:11.184421 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184168 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-oauth-serving-cert\") pod \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " Sep 09 14:44:11.184421 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184204 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-service-ca\") pod \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " Sep 09 14:44:11.184421 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184230 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-serving-cert\") pod \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " Sep 09 14:44:11.184421 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184272 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-oauth-config\") pod \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\" (UID: \"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a\") " Sep 09 14:44:11.184670 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184582 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-config" (OuterVolumeSpecName: "console-config") pod "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" (UID: "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:44:11.184670 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184601 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" (UID: "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:44:11.184670 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184628 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" (UID: "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:44:11.184926 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.184904 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-service-ca" (OuterVolumeSpecName: "service-ca") pod "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" (UID: "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:44:11.186339 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.186316 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-kube-api-access-v7v7h" (OuterVolumeSpecName: "kube-api-access-v7v7h") pod "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" (UID: "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a"). InnerVolumeSpecName "kube-api-access-v7v7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:44:11.186429 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.186338 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" (UID: "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:44:11.186429 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.186376 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" (UID: "9cdcf315-f0cd-4379-b8bb-a47b5fd9249a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:44:11.251707 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.251675 2578 generic.go:356] "Generic (PLEG): container finished" podID="9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" containerID="c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e" exitCode=0 Sep 09 14:44:11.251849 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.251743 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cdc4676-2xnqt" Sep 09 14:44:11.251849 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.251756 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cdc4676-2xnqt" event={"ID":"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a","Type":"ContainerDied","Data":"c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e"} Sep 09 14:44:11.251849 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.251802 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cdc4676-2xnqt" event={"ID":"9cdcf315-f0cd-4379-b8bb-a47b5fd9249a","Type":"ContainerDied","Data":"e6b78d195a9a7d978cd1f1468525f247a121a89727c376eb4c5027298e30ba0d"} Sep 09 14:44:11.251849 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.251823 2578 scope.go:117] "RemoveContainer" containerID="c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e" Sep 09 14:44:11.260310 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.260295 2578 scope.go:117] "RemoveContainer" containerID="c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e" Sep 09 14:44:11.261731 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:11.261642 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e\": container with ID starting with c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e not found: ID does not exist" containerID="c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e" Sep 09 14:44:11.261731 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.261678 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e"} err="failed to get container status \"c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e\": rpc error: code = NotFound desc = could not find container \"c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e\": container with ID starting with c8fb651aaa41b6a64854c6a6e932775cc6a8e5fc1d91b390b6202812db84ca0e not found: ID does not exist" Sep 09 14:44:11.273492 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.273467 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84cdc4676-2xnqt"] Sep 09 14:44:11.277991 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.277972 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84cdc4676-2xnqt"] Sep 09 14:44:11.284870 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.284850 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-config\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:11.284870 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.284871 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-oauth-serving-cert\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:11.284992 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.284882 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-service-ca\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:11.284992 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.284891 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-serving-cert\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:11.284992 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.284899 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-console-oauth-config\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:11.284992 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.284912 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7v7h\" (UniqueName: \"kubernetes.io/projected/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-kube-api-access-v7v7h\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:11.284992 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:11.284922 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a-trusted-ca-bundle\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:12.981412 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:12.981376 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" path="/var/lib/kubelet/pods/9cdcf315-f0cd-4379-b8bb-a47b5fd9249a/volumes" Sep 09 14:44:24.445552 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.445515 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h"] Sep 09 14:44:24.445990 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.445809 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="registry" containerID="cri-o://bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc" gracePeriod=30 Sep 09 14:44:24.445990 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.445924 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="ui" containerID="cri-o://0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f" gracePeriod=30 Sep 09 14:44:24.445990 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.445953 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="offline" containerID="cri-o://c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50" gracePeriod=30 Sep 09 14:44:24.446189 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.445998 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="online" containerID="cri-o://ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206" gracePeriod=30 Sep 09 14:44:24.974193 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.974157 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554"] Sep 09 14:44:24.974483 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.974470 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" containerName="console" Sep 09 14:44:24.974537 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.974485 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" containerName="console" Sep 09 14:44:24.974572 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.974542 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cdcf315-f0cd-4379-b8bb-a47b5fd9249a" containerName="console" Sep 09 14:44:24.979032 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.979008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:24.983556 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:24.983531 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-pl67m\"" Sep 09 14:44:25.002561 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.002527 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554"] Sep 09 14:44:25.084793 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.084758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-ui-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.084793 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.084793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-offline-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.084983 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.084819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-registry-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.084983 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.084947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvzt5\" (UniqueName: \"kubernetes.io/projected/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-kube-api-access-xvzt5\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.085055 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.085002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-online-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.085055 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.085020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-feast-data\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.185928 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.185898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvzt5\" (UniqueName: \"kubernetes.io/projected/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-kube-api-access-xvzt5\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.186028 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.185950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-online-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.186074 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.186035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-feast-data\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.186074 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.186064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-ui-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.186156 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.186081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-offline-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.186156 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.186096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-registry-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.186247 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:25.186209 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 14:44:25.186336 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:25.186284 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-online-tls podName:abbce629-98fe-4c29-8f9a-e3b8ec2d1048 nodeName:}" failed. No retries permitted until 2026-09-09 14:44:25.686264384 +0000 UTC m=+665.353070416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-online-tls") pod "feast-simple-feast-setup-58fc98d76c-xj554" (UID: "abbce629-98fe-4c29-8f9a-e3b8ec2d1048") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 14:44:25.186336 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:25.186210 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 14:44:25.186465 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:25.186406 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-ui-tls podName:abbce629-98fe-4c29-8f9a-e3b8ec2d1048 nodeName:}" failed. No retries permitted until 2026-09-09 14:44:25.686383965 +0000 UTC m=+665.353189991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-ui-tls") pod "feast-simple-feast-setup-58fc98d76c-xj554" (UID: "abbce629-98fe-4c29-8f9a-e3b8ec2d1048") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 14:44:25.186465 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.186406 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-feast-data\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.188388 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.188338 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-registry-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.188534 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.188516 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-offline-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.193887 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.193862 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvzt5\" (UniqueName: \"kubernetes.io/projected/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-kube-api-access-xvzt5\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.310049 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.310018 2578 generic.go:356] "Generic (PLEG): container finished" podID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerID="0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f" exitCode=0 Sep 09 14:44:25.310049 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.310042 2578 generic.go:356] "Generic (PLEG): container finished" podID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerID="ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206" exitCode=0 Sep 09 14:44:25.310217 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.310090 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerDied","Data":"0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f"} Sep 09 14:44:25.310217 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.310130 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerDied","Data":"ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206"} Sep 09 14:44:25.690909 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.690827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-online-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.690909 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.690872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-ui-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.693219 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.693198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-ui-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.693357 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.693323 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-online-tls\") pod \"feast-simple-feast-setup-58fc98d76c-xj554\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:25.891228 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:25.891192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:26.165536 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:26.165509 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554"] Sep 09 14:44:26.166342 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:44:26.166314 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabbce629_98fe_4c29_8f9a_e3b8ec2d1048.slice/crio-0b8803291e3aa654c61ef1946334034e9db5943e876859a0b5ebf293052a5ae4 WatchSource:0}: Error finding container 0b8803291e3aa654c61ef1946334034e9db5943e876859a0b5ebf293052a5ae4: Status 404 returned error can't find the container with id 0b8803291e3aa654c61ef1946334034e9db5943e876859a0b5ebf293052a5ae4 Sep 09 14:44:26.316256 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:26.316216 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerStarted","Data":"435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda"} Sep 09 14:44:26.316256 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:26.316264 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerStarted","Data":"0b8803291e3aa654c61ef1946334034e9db5943e876859a0b5ebf293052a5ae4"} Sep 09 14:44:31.335103 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:31.335068 2578 generic.go:356] "Generic (PLEG): container finished" podID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerID="435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda" exitCode=0 Sep 09 14:44:31.335605 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:31.335146 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerDied","Data":"435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda"} Sep 09 14:44:32.340740 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:32.340701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerStarted","Data":"9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683"} Sep 09 14:44:33.074317 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:33.074250 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 14:44:33.074514 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:33.074269 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 14:44:37.360397 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:37.360292 2578 generic.go:356] "Generic (PLEG): container finished" podID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerID="9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683" exitCode=0 Sep 09 14:44:37.360397 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:37.360357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerDied","Data":"9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683"} Sep 09 14:44:38.370749 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:38.370706 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerStarted","Data":"b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2"} Sep 09 14:44:38.371215 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:38.370771 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerStarted","Data":"3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef"} Sep 09 14:44:38.371215 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:38.370791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerStarted","Data":"bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54"} Sep 09 14:44:38.371215 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:38.370809 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerStarted","Data":"fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495"} Sep 09 14:44:38.403658 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:38.403590 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podStartSLOduration=14.403570731 podStartE2EDuration="14.403570731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:44:38.402534208 +0000 UTC m=+678.069340277" watchObservedRunningTime="2026-09-09 14:44:38.403570731 +0000 UTC m=+678.070376777" Sep 09 14:44:40.892003 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:40.891963 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:40.892601 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:40.892582 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:40.892725 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:40.892713 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:40.893225 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:40.893099 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:40.893903 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:40.893865 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 09 14:44:40.894017 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:40.893888 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 09 14:44:40.894017 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:40.893905 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 09 14:44:40.894128 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:40.894011 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 14:44:43.074377 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:43.074313 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 14:44:43.075115 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:43.074394 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 14:44:43.891828 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:43.891780 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 09 14:44:43.892024 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:43.891776 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 09 14:44:43.892024 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:43.891897 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 09 14:44:43.892024 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:43.891776 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 14:44:45.892310 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:45.892269 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:45.892787 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:45.892358 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:45.892787 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:45.892400 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:45.892787 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:45.892421 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:46.893875 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:46.893837 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:46.894322 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:46.894075 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:46.894429 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:46.894333 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:46.902021 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:46.901996 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:47.418412 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:47.418383 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:47.418721 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:47.418694 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:47.418816 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:47.418769 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:47.421160 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:47.421138 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:44:53.073783 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:53.073728 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 09 14:44:53.074211 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:53.073740 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 09 14:44:53.074211 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:53.073854 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:44:53.074211 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:53.074029 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:44:55.091432 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.091407 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:44:55.109510 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.109484 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f912c6-51fc-4d74-a347-01c8ca464ce5-feast-data\") pod \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " Sep 09 14:44:55.109632 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.109528 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-online-tls\") pod \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " Sep 09 14:44:55.109632 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.109589 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-registry-tls\") pod \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " Sep 09 14:44:55.109632 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.109616 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-ui-tls\") pod \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " Sep 09 14:44:55.109778 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.109642 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw6g2\" (UniqueName: \"kubernetes.io/projected/a8f912c6-51fc-4d74-a347-01c8ca464ce5-kube-api-access-xw6g2\") pod \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " Sep 09 14:44:55.109778 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.109669 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-offline-tls\") pod \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\" (UID: \"a8f912c6-51fc-4d74-a347-01c8ca464ce5\") " Sep 09 14:44:55.110040 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.110004 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f912c6-51fc-4d74-a347-01c8ca464ce5-feast-data" (OuterVolumeSpecName: "feast-data") pod "a8f912c6-51fc-4d74-a347-01c8ca464ce5" (UID: "a8f912c6-51fc-4d74-a347-01c8ca464ce5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:44:55.111910 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.111888 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-online-tls" (OuterVolumeSpecName: "online-tls") pod "a8f912c6-51fc-4d74-a347-01c8ca464ce5" (UID: "a8f912c6-51fc-4d74-a347-01c8ca464ce5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:44:55.112116 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.112086 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a8f912c6-51fc-4d74-a347-01c8ca464ce5" (UID: "a8f912c6-51fc-4d74-a347-01c8ca464ce5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:44:55.112116 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.112106 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a8f912c6-51fc-4d74-a347-01c8ca464ce5" (UID: "a8f912c6-51fc-4d74-a347-01c8ca464ce5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:44:55.112237 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.112126 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a8f912c6-51fc-4d74-a347-01c8ca464ce5" (UID: "a8f912c6-51fc-4d74-a347-01c8ca464ce5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:44:55.112237 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.112157 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f912c6-51fc-4d74-a347-01c8ca464ce5-kube-api-access-xw6g2" (OuterVolumeSpecName: "kube-api-access-xw6g2") pod "a8f912c6-51fc-4d74-a347-01c8ca464ce5" (UID: "a8f912c6-51fc-4d74-a347-01c8ca464ce5"). InnerVolumeSpecName "kube-api-access-xw6g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:44:55.211059 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.211025 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-ui-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:55.211059 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.211053 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw6g2\" (UniqueName: \"kubernetes.io/projected/a8f912c6-51fc-4d74-a347-01c8ca464ce5-kube-api-access-xw6g2\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:55.211059 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.211064 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-offline-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:55.211294 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.211073 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f912c6-51fc-4d74-a347-01c8ca464ce5-feast-data\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:55.211294 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.211081 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-online-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:55.211294 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.211088 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a8f912c6-51fc-4d74-a347-01c8ca464ce5-registry-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:44:55.452171 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.452139 2578 generic.go:356] "Generic (PLEG): container finished" podID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerID="c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50" exitCode=137 Sep 09 14:44:55.452171 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.452163 2578 generic.go:356] "Generic (PLEG): container finished" podID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerID="bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc" exitCode=137 Sep 09 14:44:55.452393 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.452202 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" Sep 09 14:44:55.452393 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.452224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerDied","Data":"c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50"} Sep 09 14:44:55.452393 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.452275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerDied","Data":"bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc"} Sep 09 14:44:55.452393 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.452292 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h" event={"ID":"a8f912c6-51fc-4d74-a347-01c8ca464ce5","Type":"ContainerDied","Data":"698d18a4d4b8b4f28f085e2a0278171c4bc5de2405917b16146f2d7a1d758aaa"} Sep 09 14:44:55.452393 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.452311 2578 scope.go:117] "RemoveContainer" containerID="0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f" Sep 09 14:44:55.466507 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.466486 2578 scope.go:117] "RemoveContainer" containerID="c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50" Sep 09 14:44:55.474174 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.474157 2578 scope.go:117] "RemoveContainer" containerID="ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206" Sep 09 14:44:55.479604 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.479584 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h"] Sep 09 14:44:55.482724 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.482708 2578 scope.go:117] "RemoveContainer" containerID="bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc" Sep 09 14:44:55.483440 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.483421 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-7bp5h"] Sep 09 14:44:55.489682 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.489666 2578 scope.go:117] "RemoveContainer" containerID="a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d" Sep 09 14:44:55.496173 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.496158 2578 scope.go:117] "RemoveContainer" containerID="7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8" Sep 09 14:44:55.502830 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.502810 2578 scope.go:117] "RemoveContainer" containerID="0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f" Sep 09 14:44:55.503066 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:55.503046 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f\": container with ID starting with 0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f not found: ID does not exist" containerID="0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f" Sep 09 14:44:55.503131 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.503077 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f"} err="failed to get container status \"0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f\": rpc error: code = NotFound desc = could not find container \"0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f\": container with ID starting with 0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f not found: ID does not exist" Sep 09 14:44:55.503131 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.503102 2578 scope.go:117] "RemoveContainer" containerID="c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50" Sep 09 14:44:55.503387 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:55.503366 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50\": container with ID starting with c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50 not found: ID does not exist" containerID="c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50" Sep 09 14:44:55.503482 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.503391 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50"} err="failed to get container status \"c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50\": rpc error: code = NotFound desc = could not find container \"c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50\": container with ID starting with c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50 not found: ID does not exist" Sep 09 14:44:55.503482 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.503407 2578 scope.go:117] "RemoveContainer" containerID="ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206" Sep 09 14:44:55.503622 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:55.503607 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206\": container with ID starting with ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206 not found: ID does not exist" containerID="ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206" Sep 09 14:44:55.503658 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.503627 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206"} err="failed to get container status \"ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206\": rpc error: code = NotFound desc = could not find container \"ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206\": container with ID starting with ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206 not found: ID does not exist" Sep 09 14:44:55.503658 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.503640 2578 scope.go:117] "RemoveContainer" containerID="bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc" Sep 09 14:44:55.503874 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:55.503860 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc\": container with ID starting with bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc not found: ID does not exist" containerID="bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc" Sep 09 14:44:55.503913 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.503877 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc"} err="failed to get container status \"bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc\": rpc error: code = NotFound desc = could not find container \"bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc\": container with ID starting with bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc not found: ID does not exist" Sep 09 14:44:55.503913 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.503890 2578 scope.go:117] "RemoveContainer" containerID="a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d" Sep 09 14:44:55.504090 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:55.504069 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d\": container with ID starting with a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d not found: ID does not exist" containerID="a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d" Sep 09 14:44:55.504155 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.504099 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d"} err="failed to get container status \"a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d\": rpc error: code = NotFound desc = could not find container \"a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d\": container with ID starting with a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d not found: ID does not exist" Sep 09 14:44:55.504155 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.504120 2578 scope.go:117] "RemoveContainer" containerID="7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8" Sep 09 14:44:55.504386 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:44:55.504366 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8\": container with ID starting with 7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8 not found: ID does not exist" containerID="7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8" Sep 09 14:44:55.504421 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.504393 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8"} err="failed to get container status \"7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8\": rpc error: code = NotFound desc = could not find container \"7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8\": container with ID starting with 7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8 not found: ID does not exist" Sep 09 14:44:55.504421 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.504410 2578 scope.go:117] "RemoveContainer" containerID="0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f" Sep 09 14:44:55.504648 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.504628 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f"} err="failed to get container status \"0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f\": rpc error: code = NotFound desc = could not find container \"0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f\": container with ID starting with 0b8c70d38bc031313fc7c4628e5efede0f7168e232a9d9f0dbe37c673fab315f not found: ID does not exist" Sep 09 14:44:55.504716 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.504649 2578 scope.go:117] "RemoveContainer" containerID="c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50" Sep 09 14:44:55.504875 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.504857 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50"} err="failed to get container status \"c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50\": rpc error: code = NotFound desc = could not find container \"c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50\": container with ID starting with c4c1434d2e4f493bc6548f690c5d265392069b37b43d55bb8d7039d9dc3ecb50 not found: ID does not exist" Sep 09 14:44:55.504919 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.504876 2578 scope.go:117] "RemoveContainer" containerID="ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206" Sep 09 14:44:55.505111 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.505093 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206"} err="failed to get container status \"ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206\": rpc error: code = NotFound desc = could not find container \"ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206\": container with ID starting with ed5519a6de556ce25814867ae443fbb08d5dba8b5abe6301217cc165c92a1206 not found: ID does not exist" Sep 09 14:44:55.505178 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.505112 2578 scope.go:117] "RemoveContainer" containerID="bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc" Sep 09 14:44:55.505326 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.505311 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc"} err="failed to get container status \"bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc\": rpc error: code = NotFound desc = could not find container \"bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc\": container with ID starting with bf29d594f7d3fbbe2604e79f7a49a0854a7709f57f2712a000024f1cc1f775dc not found: ID does not exist" Sep 09 14:44:55.505444 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.505325 2578 scope.go:117] "RemoveContainer" containerID="a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d" Sep 09 14:44:55.505526 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.505512 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d"} err="failed to get container status \"a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d\": rpc error: code = NotFound desc = could not find container \"a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d\": container with ID starting with a0afc4bd173b0a27c7db379e0d7dbe14e85fbcb56e1b3cd35368e36e4173094d not found: ID does not exist" Sep 09 14:44:55.505526 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.505525 2578 scope.go:117] "RemoveContainer" containerID="7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8" Sep 09 14:44:55.505743 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:55.505722 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8"} err="failed to get container status \"7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8\": rpc error: code = NotFound desc = could not find container \"7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8\": container with ID starting with 7f2d31b884b7f3b446bb33406ddff162dab79402c920b340c9c9c7542abea3c8 not found: ID does not exist" Sep 09 14:44:56.981032 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:44:56.980997 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" path="/var/lib/kubelet/pods/a8f912c6-51fc-4d74-a347-01c8ca464ce5/volumes" Sep 09 14:45:49.868152 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868114 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7"] Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868412 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="feast-init" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868426 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="feast-init" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868436 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="feast-apply" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868442 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="feast-apply" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868451 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="online" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868457 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="online" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868464 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="ui" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868468 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="ui" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868477 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="registry" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868482 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="registry" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868493 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="offline" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868498 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="offline" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868542 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="online" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868551 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="offline" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868558 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="ui" Sep 09 14:45:49.868589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.868563 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f912c6-51fc-4d74-a347-01c8ca464ce5" containerName="registry" Sep 09 14:45:49.871698 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.871678 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:49.874205 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.874180 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 14:45:49.874330 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.874180 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 14:45:49.874330 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.874235 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 09 14:45:49.874330 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.874316 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 14:45:49.874524 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.874374 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 14:45:49.877045 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.877023 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 14:45:49.877144 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.877125 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 14:45:49.877331 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.877311 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-9lfmx\"" Sep 09 14:45:49.887766 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:49.887744 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7"] Sep 09 14:45:50.017222 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.017188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-online-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.017430 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.017233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-offline-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.017430 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.017331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8218b3b5-7b24-4d46-9c72-cbccb6200106-feast-data\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.017430 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.017404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8218b3b5-7b24-4d46-9c72-cbccb6200106-registry-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.017585 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.017431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzck\" (UniqueName: \"kubernetes.io/projected/8218b3b5-7b24-4d46-9c72-cbccb6200106-kube-api-access-ghzck\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.017585 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.017472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-ui-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.118535 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.118464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-online-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.118535 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.118504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-offline-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.118535 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.118526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8218b3b5-7b24-4d46-9c72-cbccb6200106-feast-data\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.118750 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.118547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8218b3b5-7b24-4d46-9c72-cbccb6200106-registry-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.118750 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.118564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzck\" (UniqueName: \"kubernetes.io/projected/8218b3b5-7b24-4d46-9c72-cbccb6200106-kube-api-access-ghzck\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.118750 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.118588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-ui-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.118750 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:45:50.118622 2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 14:45:50.118750 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:45:50.118688 2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 14:45:50.118750 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:45:50.118699 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-online-tls podName:8218b3b5-7b24-4d46-9c72-cbccb6200106 nodeName:}" failed. No retries permitted until 2026-09-09 14:45:50.618678114 +0000 UTC m=+750.285484140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-online-tls") pod "feast-simple-feast-remote-setup-65b4c766c9-cz6w7" (UID: "8218b3b5-7b24-4d46-9c72-cbccb6200106") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 14:45:50.118750 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:45:50.118742 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-ui-tls podName:8218b3b5-7b24-4d46-9c72-cbccb6200106 nodeName:}" failed. No retries permitted until 2026-09-09 14:45:50.618727441 +0000 UTC m=+750.285533468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-ui-tls") pod "feast-simple-feast-remote-setup-65b4c766c9-cz6w7" (UID: "8218b3b5-7b24-4d46-9c72-cbccb6200106") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 14:45:50.119044 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.118918 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8218b3b5-7b24-4d46-9c72-cbccb6200106-feast-data\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.119305 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.119287 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8218b3b5-7b24-4d46-9c72-cbccb6200106-registry-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.121061 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.121036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-offline-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.127479 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.127457 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzck\" (UniqueName: \"kubernetes.io/projected/8218b3b5-7b24-4d46-9c72-cbccb6200106-kube-api-access-ghzck\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.624264 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.624227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-online-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.624432 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.624286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-ui-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.626761 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.626731 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-ui-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.626868 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.626783 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-online-tls\") pod \"feast-simple-feast-remote-setup-65b4c766c9-cz6w7\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:50.780300 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:50.780267 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:45:51.058142 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:51.058118 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7"] Sep 09 14:45:51.059715 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:45:51.059690 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8218b3b5_7b24_4d46_9c72_cbccb6200106.slice/crio-676633c5cbe36cdffb5e963d0adb5e4b728b84f82d2e27134819a720d6824971 WatchSource:0}: Error finding container 676633c5cbe36cdffb5e963d0adb5e4b728b84f82d2e27134819a720d6824971: Status 404 returned error can't find the container with id 676633c5cbe36cdffb5e963d0adb5e4b728b84f82d2e27134819a720d6824971 Sep 09 14:45:51.661207 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:51.661172 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerStarted","Data":"66a24ac1636ae7ff29d65d19854bcf530479a9b04a8a114d54faab688dd219c8"} Sep 09 14:45:51.661467 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:51.661220 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerStarted","Data":"676633c5cbe36cdffb5e963d0adb5e4b728b84f82d2e27134819a720d6824971"} Sep 09 14:45:56.682178 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:56.682143 2578 generic.go:356] "Generic (PLEG): container finished" podID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerID="66a24ac1636ae7ff29d65d19854bcf530479a9b04a8a114d54faab688dd219c8" exitCode=0 Sep 09 14:45:56.682575 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:56.682190 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerDied","Data":"66a24ac1636ae7ff29d65d19854bcf530479a9b04a8a114d54faab688dd219c8"} Sep 09 14:45:57.688065 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:45:57.688030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerStarted","Data":"352a3ce6e70569e1a6ae74e0fa4f8d56cec4d2629185720c89ea88bac7ecda66"} Sep 09 14:46:21.779698 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:21.779661 2578 generic.go:356] "Generic (PLEG): container finished" podID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerID="352a3ce6e70569e1a6ae74e0fa4f8d56cec4d2629185720c89ea88bac7ecda66" exitCode=0 Sep 09 14:46:21.780056 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:21.779735 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerDied","Data":"352a3ce6e70569e1a6ae74e0fa4f8d56cec4d2629185720c89ea88bac7ecda66"} Sep 09 14:46:21.783127 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:21.783107 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 09 14:46:22.788601 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:22.788557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerStarted","Data":"c9764adda678e0f2b45551bb18807116b48021d6d60faf0b7f5b2894957b5f0a"} Sep 09 14:46:22.789182 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:22.788617 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerStarted","Data":"ca2950b05fa50b28ddc7b22779afac159ae9d835bbe7568893f4de7dd560f38e"} Sep 09 14:46:22.789182 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:22.788637 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerStarted","Data":"f0c608c698e725611e2d91c34443391946b0b7f1fdbf1939a45308eec0cc8d56"} Sep 09 14:46:22.815418 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:22.815360 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podStartSLOduration=33.8153286 podStartE2EDuration="33.8153286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:46:22.813984639 +0000 UTC m=+782.480790689" watchObservedRunningTime="2026-09-09 14:46:22.8153286 +0000 UTC m=+782.482134647" Sep 09 14:46:23.780587 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:23.780541 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:23.780789 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:23.780606 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:23.780789 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:23.780626 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:23.782555 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:23.782521 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Sep 09 14:46:23.782690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:23.782577 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 14:46:23.782690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:23.782521 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Sep 09 14:46:26.781059 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:26.781007 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Sep 09 14:46:26.781059 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:26.781027 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Sep 09 14:46:26.781687 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:26.781037 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 14:46:29.781473 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:29.781417 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Sep 09 14:46:29.782060 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:29.782027 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:29.782404 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:29.782379 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:29.782801 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:29.782781 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:29.803694 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:29.803652 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:29.825408 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:29.825373 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:29.904753 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:29.904719 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:30.781127 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:30.781090 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:32.781591 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:32.781553 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:46:32.835396 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:46:32.835342 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:47:34.143371 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:34.143312 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7"] Sep 09 14:47:34.143940 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:34.143601 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="online" containerID="cri-o://f0c608c698e725611e2d91c34443391946b0b7f1fdbf1939a45308eec0cc8d56" gracePeriod=30 Sep 09 14:47:34.143940 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:34.143799 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" containerID="cri-o://c9764adda678e0f2b45551bb18807116b48021d6d60faf0b7f5b2894957b5f0a" gracePeriod=30 Sep 09 14:47:34.143940 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:34.143840 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="offline" containerID="cri-o://ca2950b05fa50b28ddc7b22779afac159ae9d835bbe7568893f4de7dd560f38e" gracePeriod=30 Sep 09 14:47:34.248694 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:34.248664 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554"] Sep 09 14:47:34.249487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:34.248966 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="registry" containerID="cri-o://fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495" gracePeriod=30 Sep 09 14:47:34.249487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:34.249017 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="offline" containerID="cri-o://3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef" gracePeriod=30 Sep 09 14:47:34.249487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:34.249029 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="online" containerID="cri-o://bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54" gracePeriod=30 Sep 09 14:47:34.249487 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:34.249011 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="ui" containerID="cri-o://b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2" gracePeriod=30 Sep 09 14:47:35.070312 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:35.070286 2578 generic.go:356] "Generic (PLEG): container finished" podID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerID="c9764adda678e0f2b45551bb18807116b48021d6d60faf0b7f5b2894957b5f0a" exitCode=0 Sep 09 14:47:35.070312 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:35.070309 2578 generic.go:356] "Generic (PLEG): container finished" podID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerID="f0c608c698e725611e2d91c34443391946b0b7f1fdbf1939a45308eec0cc8d56" exitCode=0 Sep 09 14:47:35.070458 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:35.070366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerDied","Data":"c9764adda678e0f2b45551bb18807116b48021d6d60faf0b7f5b2894957b5f0a"} Sep 09 14:47:35.070458 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:35.070405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerDied","Data":"f0c608c698e725611e2d91c34443391946b0b7f1fdbf1939a45308eec0cc8d56"} Sep 09 14:47:35.073263 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:35.073242 2578 generic.go:356] "Generic (PLEG): container finished" podID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerID="b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2" exitCode=0 Sep 09 14:47:35.073263 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:35.073263 2578 generic.go:356] "Generic (PLEG): container finished" podID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerID="bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54" exitCode=0 Sep 09 14:47:35.073423 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:35.073294 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerDied","Data":"b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2"} Sep 09 14:47:35.073423 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:35.073329 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerDied","Data":"bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54"} Sep 09 14:47:37.418370 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:37.418294 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 14:47:37.418819 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:37.418304 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 09 14:47:39.826191 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:39.826139 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 14:47:42.836047 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:42.835999 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Sep 09 14:47:47.418772 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:47.418707 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 09 14:47:47.419215 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:47.418710 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 14:47:49.825926 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:49.825875 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 14:47:52.835510 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:52.835458 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Sep 09 14:47:57.418178 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:57.418129 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 09 14:47:57.418178 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:57.418170 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 09 14:47:57.418690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:57.418269 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:47:57.418690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:57.418339 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:47:59.825959 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:59.825911 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Sep 09 14:47:59.826467 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:47:59.826055 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:48:02.835957 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:02.835909 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Sep 09 14:48:02.836393 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:02.836062 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:48:04.187276 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.187246 2578 generic.go:356] "Generic (PLEG): container finished" podID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerID="ca2950b05fa50b28ddc7b22779afac159ae9d835bbe7568893f4de7dd560f38e" exitCode=137 Sep 09 14:48:04.187630 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.187317 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerDied","Data":"ca2950b05fa50b28ddc7b22779afac159ae9d835bbe7568893f4de7dd560f38e"} Sep 09 14:48:04.799769 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.799744 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:48:04.857453 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.857419 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8218b3b5-7b24-4d46-9c72-cbccb6200106-registry-tls\") pod \"8218b3b5-7b24-4d46-9c72-cbccb6200106\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " Sep 09 14:48:04.857625 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.857475 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghzck\" (UniqueName: \"kubernetes.io/projected/8218b3b5-7b24-4d46-9c72-cbccb6200106-kube-api-access-ghzck\") pod \"8218b3b5-7b24-4d46-9c72-cbccb6200106\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " Sep 09 14:48:04.857625 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.857600 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-offline-tls\") pod \"8218b3b5-7b24-4d46-9c72-cbccb6200106\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " Sep 09 14:48:04.857738 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.857656 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8218b3b5-7b24-4d46-9c72-cbccb6200106-feast-data\") pod \"8218b3b5-7b24-4d46-9c72-cbccb6200106\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " Sep 09 14:48:04.857738 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.857682 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-ui-tls\") pod \"8218b3b5-7b24-4d46-9c72-cbccb6200106\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " Sep 09 14:48:04.857738 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.857710 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-online-tls\") pod \"8218b3b5-7b24-4d46-9c72-cbccb6200106\" (UID: \"8218b3b5-7b24-4d46-9c72-cbccb6200106\") " Sep 09 14:48:04.857948 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.857825 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8218b3b5-7b24-4d46-9c72-cbccb6200106-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8218b3b5-7b24-4d46-9c72-cbccb6200106" (UID: "8218b3b5-7b24-4d46-9c72-cbccb6200106"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 14:48:04.858085 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.858046 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8218b3b5-7b24-4d46-9c72-cbccb6200106-registry-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:04.858256 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.858217 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8218b3b5-7b24-4d46-9c72-cbccb6200106-feast-data" (OuterVolumeSpecName: "feast-data") pod "8218b3b5-7b24-4d46-9c72-cbccb6200106" (UID: "8218b3b5-7b24-4d46-9c72-cbccb6200106"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:48:04.860176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.860150 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-online-tls" (OuterVolumeSpecName: "online-tls") pod "8218b3b5-7b24-4d46-9c72-cbccb6200106" (UID: "8218b3b5-7b24-4d46-9c72-cbccb6200106"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:48:04.860176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.860162 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8218b3b5-7b24-4d46-9c72-cbccb6200106-kube-api-access-ghzck" (OuterVolumeSpecName: "kube-api-access-ghzck") pod "8218b3b5-7b24-4d46-9c72-cbccb6200106" (UID: "8218b3b5-7b24-4d46-9c72-cbccb6200106"). InnerVolumeSpecName "kube-api-access-ghzck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:48:04.860399 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.860205 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "8218b3b5-7b24-4d46-9c72-cbccb6200106" (UID: "8218b3b5-7b24-4d46-9c72-cbccb6200106"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:48:04.860399 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.860225 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "8218b3b5-7b24-4d46-9c72-cbccb6200106" (UID: "8218b3b5-7b24-4d46-9c72-cbccb6200106"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:48:04.891938 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.891911 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:48:04.958695 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.958607 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-registry-tls\") pod \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " Sep 09 14:48:04.958695 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.958659 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-online-tls\") pod \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " Sep 09 14:48:04.958695 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.958691 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-ui-tls\") pod \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " Sep 09 14:48:04.958915 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.958894 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvzt5\" (UniqueName: \"kubernetes.io/projected/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-kube-api-access-xvzt5\") pod \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " Sep 09 14:48:04.958962 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.958951 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-feast-data\") pod \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " Sep 09 14:48:04.959142 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.959062 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-offline-tls\") pod \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\" (UID: \"abbce629-98fe-4c29-8f9a-e3b8ec2d1048\") " Sep 09 14:48:04.959309 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.959290 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-offline-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:04.959469 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.959441 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-feast-data" (OuterVolumeSpecName: "feast-data") pod "abbce629-98fe-4c29-8f9a-e3b8ec2d1048" (UID: "abbce629-98fe-4c29-8f9a-e3b8ec2d1048"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:48:04.959559 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.959446 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8218b3b5-7b24-4d46-9c72-cbccb6200106-feast-data\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:04.959559 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.959494 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-ui-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:04.959559 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.959509 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8218b3b5-7b24-4d46-9c72-cbccb6200106-online-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:04.959559 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.959523 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzck\" (UniqueName: \"kubernetes.io/projected/8218b3b5-7b24-4d46-9c72-cbccb6200106-kube-api-access-ghzck\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:04.961048 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.961013 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-online-tls" (OuterVolumeSpecName: "online-tls") pod "abbce629-98fe-4c29-8f9a-e3b8ec2d1048" (UID: "abbce629-98fe-4c29-8f9a-e3b8ec2d1048"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:48:04.961234 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.961217 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "abbce629-98fe-4c29-8f9a-e3b8ec2d1048" (UID: "abbce629-98fe-4c29-8f9a-e3b8ec2d1048"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:48:04.961308 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.961239 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "abbce629-98fe-4c29-8f9a-e3b8ec2d1048" (UID: "abbce629-98fe-4c29-8f9a-e3b8ec2d1048"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:48:04.961308 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.961237 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-kube-api-access-xvzt5" (OuterVolumeSpecName: "kube-api-access-xvzt5") pod "abbce629-98fe-4c29-8f9a-e3b8ec2d1048" (UID: "abbce629-98fe-4c29-8f9a-e3b8ec2d1048"). InnerVolumeSpecName "kube-api-access-xvzt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:48:04.961403 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:04.961316 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "abbce629-98fe-4c29-8f9a-e3b8ec2d1048" (UID: "abbce629-98fe-4c29-8f9a-e3b8ec2d1048"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:48:05.060096 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.060059 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-registry-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:05.060096 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.060096 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-online-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:05.060308 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.060108 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-ui-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:05.060308 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.060122 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvzt5\" (UniqueName: \"kubernetes.io/projected/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-kube-api-access-xvzt5\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:05.060308 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.060135 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-feast-data\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:05.060308 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.060149 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/abbce629-98fe-4c29-8f9a-e3b8ec2d1048-offline-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:48:05.193945 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.193911 2578 generic.go:356] "Generic (PLEG): container finished" podID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerID="3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef" exitCode=137 Sep 09 14:48:05.193945 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.193938 2578 generic.go:356] "Generic (PLEG): container finished" podID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerID="fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495" exitCode=137 Sep 09 14:48:05.194492 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.193975 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" Sep 09 14:48:05.194492 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.193997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerDied","Data":"3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef"} Sep 09 14:48:05.194492 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.194036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerDied","Data":"fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495"} Sep 09 14:48:05.194492 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.194054 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554" event={"ID":"abbce629-98fe-4c29-8f9a-e3b8ec2d1048","Type":"ContainerDied","Data":"0b8803291e3aa654c61ef1946334034e9db5943e876859a0b5ebf293052a5ae4"} Sep 09 14:48:05.194492 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.194080 2578 scope.go:117] "RemoveContainer" containerID="b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2" Sep 09 14:48:05.197316 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.197290 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" event={"ID":"8218b3b5-7b24-4d46-9c72-cbccb6200106","Type":"ContainerDied","Data":"676633c5cbe36cdffb5e963d0adb5e4b728b84f82d2e27134819a720d6824971"} Sep 09 14:48:05.197440 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.197365 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7" Sep 09 14:48:05.203045 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.203028 2578 scope.go:117] "RemoveContainer" containerID="3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef" Sep 09 14:48:05.211280 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.211261 2578 scope.go:117] "RemoveContainer" containerID="bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54" Sep 09 14:48:05.211411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.211387 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554"] Sep 09 14:48:05.215802 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.215780 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-58fc98d76c-xj554"] Sep 09 14:48:05.219747 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.219730 2578 scope.go:117] "RemoveContainer" containerID="fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495" Sep 09 14:48:05.227079 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.227061 2578 scope.go:117] "RemoveContainer" containerID="9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683" Sep 09 14:48:05.230176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.230156 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7"] Sep 09 14:48:05.233315 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.233288 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-65b4c766c9-cz6w7"] Sep 09 14:48:05.234933 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.234918 2578 scope.go:117] "RemoveContainer" containerID="435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda" Sep 09 14:48:05.242307 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.242288 2578 scope.go:117] "RemoveContainer" containerID="b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2" Sep 09 14:48:05.242598 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:05.242580 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2\": container with ID starting with b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2 not found: ID does not exist" containerID="b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2" Sep 09 14:48:05.242674 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.242604 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2"} err="failed to get container status \"b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2\": rpc error: code = NotFound desc = could not find container \"b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2\": container with ID starting with b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2 not found: ID does not exist" Sep 09 14:48:05.242674 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.242629 2578 scope.go:117] "RemoveContainer" containerID="3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef" Sep 09 14:48:05.242884 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:05.242865 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef\": container with ID starting with 3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef not found: ID does not exist" containerID="3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef" Sep 09 14:48:05.242923 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.242894 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef"} err="failed to get container status \"3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef\": rpc error: code = NotFound desc = could not find container \"3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef\": container with ID starting with 3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef not found: ID does not exist" Sep 09 14:48:05.242923 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.242912 2578 scope.go:117] "RemoveContainer" containerID="bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54" Sep 09 14:48:05.243140 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:05.243123 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54\": container with ID starting with bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54 not found: ID does not exist" containerID="bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54" Sep 09 14:48:05.243192 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.243143 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54"} err="failed to get container status \"bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54\": rpc error: code = NotFound desc = could not find container \"bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54\": container with ID starting with bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54 not found: ID does not exist" Sep 09 14:48:05.243192 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.243156 2578 scope.go:117] "RemoveContainer" containerID="fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495" Sep 09 14:48:05.243340 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:05.243325 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495\": container with ID starting with fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495 not found: ID does not exist" containerID="fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495" Sep 09 14:48:05.243340 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.243359 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495"} err="failed to get container status \"fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495\": rpc error: code = NotFound desc = could not find container \"fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495\": container with ID starting with fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495 not found: ID does not exist" Sep 09 14:48:05.243514 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.243377 2578 scope.go:117] "RemoveContainer" containerID="9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683" Sep 09 14:48:05.243647 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:05.243630 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683\": container with ID starting with 9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683 not found: ID does not exist" containerID="9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683" Sep 09 14:48:05.243692 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.243651 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683"} err="failed to get container status \"9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683\": rpc error: code = NotFound desc = could not find container \"9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683\": container with ID starting with 9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683 not found: ID does not exist" Sep 09 14:48:05.243692 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.243664 2578 scope.go:117] "RemoveContainer" containerID="435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda" Sep 09 14:48:05.243906 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:05.243888 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda\": container with ID starting with 435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda not found: ID does not exist" containerID="435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda" Sep 09 14:48:05.243953 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.243908 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda"} err="failed to get container status \"435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda\": rpc error: code = NotFound desc = could not find container \"435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda\": container with ID starting with 435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda not found: ID does not exist" Sep 09 14:48:05.243953 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.243919 2578 scope.go:117] "RemoveContainer" containerID="b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2" Sep 09 14:48:05.244122 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.244107 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2"} err="failed to get container status \"b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2\": rpc error: code = NotFound desc = could not find container \"b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2\": container with ID starting with b6c059ce9b2e1cf1a2145056888dcd437435f2c79947780293a7c0264311fcf2 not found: ID does not exist" Sep 09 14:48:05.244162 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.244122 2578 scope.go:117] "RemoveContainer" containerID="3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef" Sep 09 14:48:05.244339 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.244320 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef"} err="failed to get container status \"3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef\": rpc error: code = NotFound desc = could not find container \"3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef\": container with ID starting with 3c95ac0dabf74753c367ab2ef1e4ef9c44b25a6dea04f3a83f42754569f872ef not found: ID does not exist" Sep 09 14:48:05.244339 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.244339 2578 scope.go:117] "RemoveContainer" containerID="bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54" Sep 09 14:48:05.244574 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.244558 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54"} err="failed to get container status \"bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54\": rpc error: code = NotFound desc = could not find container \"bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54\": container with ID starting with bda314bac5a802fb826cf8099ed8146d7f869d247c16792b7aed75844cf21c54 not found: ID does not exist" Sep 09 14:48:05.244615 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.244574 2578 scope.go:117] "RemoveContainer" containerID="fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495" Sep 09 14:48:05.244803 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.244785 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495"} err="failed to get container status \"fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495\": rpc error: code = NotFound desc = could not find container \"fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495\": container with ID starting with fb4ec5b29096531d64c555497a08b3f3a79110ac3522f617b02ad55b94c0a495 not found: ID does not exist" Sep 09 14:48:05.244851 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.244804 2578 scope.go:117] "RemoveContainer" containerID="9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683" Sep 09 14:48:05.245025 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.245009 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683"} err="failed to get container status \"9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683\": rpc error: code = NotFound desc = could not find container \"9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683\": container with ID starting with 9cfa95c8101a0b3c4a2828436e34e2754522cc7e403df0c2ca24881d1859d683 not found: ID does not exist" Sep 09 14:48:05.245025 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.245025 2578 scope.go:117] "RemoveContainer" containerID="435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda" Sep 09 14:48:05.245255 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.245240 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda"} err="failed to get container status \"435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda\": rpc error: code = NotFound desc = could not find container \"435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda\": container with ID starting with 435cd0a590c5778e2d7425d2a9245e69c276f57056d837bf5a60507880c71eda not found: ID does not exist" Sep 09 14:48:05.245295 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.245256 2578 scope.go:117] "RemoveContainer" containerID="c9764adda678e0f2b45551bb18807116b48021d6d60faf0b7f5b2894957b5f0a" Sep 09 14:48:05.252125 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.252110 2578 scope.go:117] "RemoveContainer" containerID="ca2950b05fa50b28ddc7b22779afac159ae9d835bbe7568893f4de7dd560f38e" Sep 09 14:48:05.258971 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.258956 2578 scope.go:117] "RemoveContainer" containerID="f0c608c698e725611e2d91c34443391946b0b7f1fdbf1939a45308eec0cc8d56" Sep 09 14:48:05.266406 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.266387 2578 scope.go:117] "RemoveContainer" containerID="352a3ce6e70569e1a6ae74e0fa4f8d56cec4d2629185720c89ea88bac7ecda66" Sep 09 14:48:05.273217 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:05.273196 2578 scope.go:117] "RemoveContainer" containerID="66a24ac1636ae7ff29d65d19854bcf530479a9b04a8a114d54faab688dd219c8" Sep 09 14:48:06.981406 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:06.981376 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" path="/var/lib/kubelet/pods/8218b3b5-7b24-4d46-9c72-cbccb6200106/volumes" Sep 09 14:48:06.981925 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:06.981911 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" path="/var/lib/kubelet/pods/abbce629-98fe-4c29-8f9a-e3b8ec2d1048/volumes" Sep 09 14:48:17.080267 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080227 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-9c9cs"] Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080504 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="offline" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080517 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="offline" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080527 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="feast-apply" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080533 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="feast-apply" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080541 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="feast-apply" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080548 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="feast-apply" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080554 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="ui" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080559 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="ui" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080568 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="online" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080573 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="online" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080581 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="offline" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080586 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="offline" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080593 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="feast-init" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080597 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="feast-init" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080603 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="registry" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080608 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="registry" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080615 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080620 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080626 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="online" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080631 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="online" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080637 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="feast-init" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080642 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="feast-init" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080679 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="ui" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080687 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="registry" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080693 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="online" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080699 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="online" Sep 09 14:48:17.080690 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080705 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8218b3b5-7b24-4d46-9c72-cbccb6200106" containerName="offline" Sep 09 14:48:17.081541 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080712 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="ui" Sep 09 14:48:17.081541 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.080717 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbce629-98fe-4c29-8f9a-e3b8ec2d1048" containerName="offline" Sep 09 14:48:17.083860 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.083843 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:48:17.086220 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.086197 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-jfdpb\"" Sep 09 14:48:17.086570 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.086550 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 09 14:48:17.086691 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.086575 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 14:48:17.086925 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.086909 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 14:48:17.089310 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.089286 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-9c9cs"] Sep 09 14:48:17.249477 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.249445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f720a1da-bd1c-4a18-ba81-f67da99fbc01-postgresdata\") pod \"postgres-864846d4f5-9c9cs\" (UID: \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\") " pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:48:17.249694 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.249491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rjk\" (UniqueName: \"kubernetes.io/projected/f720a1da-bd1c-4a18-ba81-f67da99fbc01-kube-api-access-s7rjk\") pod \"postgres-864846d4f5-9c9cs\" (UID: \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\") " pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:48:17.350615 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.350514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rjk\" (UniqueName: \"kubernetes.io/projected/f720a1da-bd1c-4a18-ba81-f67da99fbc01-kube-api-access-s7rjk\") pod \"postgres-864846d4f5-9c9cs\" (UID: \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\") " pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:48:17.350615 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.350588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f720a1da-bd1c-4a18-ba81-f67da99fbc01-postgresdata\") pod \"postgres-864846d4f5-9c9cs\" (UID: \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\") " pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:48:17.350915 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.350899 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f720a1da-bd1c-4a18-ba81-f67da99fbc01-postgresdata\") pod \"postgres-864846d4f5-9c9cs\" (UID: \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\") " pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:48:17.359410 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.359379 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rjk\" (UniqueName: \"kubernetes.io/projected/f720a1da-bd1c-4a18-ba81-f67da99fbc01-kube-api-access-s7rjk\") pod \"postgres-864846d4f5-9c9cs\" (UID: \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\") " pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:48:17.394045 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.394005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:48:17.660502 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.660424 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-9c9cs"] Sep 09 14:48:17.663772 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:48:17.663743 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf720a1da_bd1c_4a18_ba81_f67da99fbc01.slice/crio-125e601f658f1a23b839efa6c37f0f53c23fc2adc6c1047e8f17a0120c29213f WatchSource:0}: Error finding container 125e601f658f1a23b839efa6c37f0f53c23fc2adc6c1047e8f17a0120c29213f: Status 404 returned error can't find the container with id 125e601f658f1a23b839efa6c37f0f53c23fc2adc6c1047e8f17a0120c29213f Sep 09 14:48:17.665520 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:17.665504 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 14:48:18.254991 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:18.254954 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-9c9cs" event={"ID":"f720a1da-bd1c-4a18-ba81-f67da99fbc01","Type":"ContainerStarted","Data":"125e601f658f1a23b839efa6c37f0f53c23fc2adc6c1047e8f17a0120c29213f"} Sep 09 14:48:20.799818 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:20.799762 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:48:20.801823 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:20.801799 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:48:23.751781 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:23.751758 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 09 14:48:24.279729 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:24.279692 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-9c9cs" event={"ID":"f720a1da-bd1c-4a18-ba81-f67da99fbc01","Type":"ContainerStarted","Data":"84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64"} Sep 09 14:48:24.295572 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:24.295506 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-9c9cs" podStartSLOduration=1.212058033 podStartE2EDuration="7.295485294s" totalImagesPullingTime="6.083427261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:48:24.29487351 +0000 UTC m=+903.961679557" watchObservedRunningTime="2026-09-09 14:48:24.295485294 +0000 UTC m=+903.962291343" Sep 09 14:48:27.959444 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:27.959405 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5"] Sep 09 14:48:27.965898 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:27.965864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:27.968395 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:27.968367 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 09 14:48:27.968395 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:27.968385 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 09 14:48:27.968609 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:27.968379 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 09 14:48:27.968609 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:27.968379 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-gvxnz\"" Sep 09 14:48:27.968609 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:27.968534 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 09 14:48:27.975627 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:27.975604 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5"] Sep 09 14:48:28.140767 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.140727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-online-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.140975 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.140778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-registry-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.140975 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.140845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/596f294d-8a20-4494-8dc4-e1ea20ef7153-feast-data\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.140975 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.140921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjb9p\" (UniqueName: \"kubernetes.io/projected/596f294d-8a20-4494-8dc4-e1ea20ef7153-kube-api-access-zjb9p\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.140975 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.140972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-offline-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.242370 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.242315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjb9p\" (UniqueName: \"kubernetes.io/projected/596f294d-8a20-4494-8dc4-e1ea20ef7153-kube-api-access-zjb9p\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.242561 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.242411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-offline-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.242561 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.242453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-online-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.242561 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.242478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-registry-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.242561 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.242494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/596f294d-8a20-4494-8dc4-e1ea20ef7153-feast-data\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.242764 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:28.242561 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Sep 09 14:48:28.242764 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:28.242578 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 09 14:48:28.242764 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:28.242660 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-offline-tls podName:596f294d-8a20-4494-8dc4-e1ea20ef7153 nodeName:}" failed. No retries permitted until 2026-09-09 14:48:28.742637808 +0000 UTC m=+908.409443836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-offline-tls") pod "feast-credit-scoring-5b75c44fd-hhcl5" (UID: "596f294d-8a20-4494-8dc4-e1ea20ef7153") : secret "feast-credit-scoring-offline-tls" not found Sep 09 14:48:28.242764 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:48:28.242681 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-registry-tls podName:596f294d-8a20-4494-8dc4-e1ea20ef7153 nodeName:}" failed. No retries permitted until 2026-09-09 14:48:28.74267187 +0000 UTC m=+908.409477897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-registry-tls") pod "feast-credit-scoring-5b75c44fd-hhcl5" (UID: "596f294d-8a20-4494-8dc4-e1ea20ef7153") : secret "feast-credit-scoring-registry-tls" not found Sep 09 14:48:28.242920 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.242858 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/596f294d-8a20-4494-8dc4-e1ea20ef7153-feast-data\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.244995 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.244966 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-online-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.250853 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.250830 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjb9p\" (UniqueName: \"kubernetes.io/projected/596f294d-8a20-4494-8dc4-e1ea20ef7153-kube-api-access-zjb9p\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.746998 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.746957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-offline-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.747175 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.747018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-registry-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.749654 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.749625 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-offline-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.749811 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.749659 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-registry-tls\") pod \"feast-credit-scoring-5b75c44fd-hhcl5\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:28.875446 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:28.875404 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:29.188826 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:29.188789 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5"] Sep 09 14:48:29.190430 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:48:29.190403 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596f294d_8a20_4494_8dc4_e1ea20ef7153.slice/crio-23833d2cd14912dc8da8c8031da5f7c354bf2660fbd79363a00b2b113906e37b WatchSource:0}: Error finding container 23833d2cd14912dc8da8c8031da5f7c354bf2660fbd79363a00b2b113906e37b: Status 404 returned error can't find the container with id 23833d2cd14912dc8da8c8031da5f7c354bf2660fbd79363a00b2b113906e37b Sep 09 14:48:29.300204 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:29.300163 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerStarted","Data":"c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783"} Sep 09 14:48:29.300392 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:29.300226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerStarted","Data":"23833d2cd14912dc8da8c8031da5f7c354bf2660fbd79363a00b2b113906e37b"} Sep 09 14:48:30.304810 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:30.304771 2578 generic.go:356] "Generic (PLEG): container finished" podID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerID="c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783" exitCode=0 Sep 09 14:48:30.305200 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:30.304828 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerDied","Data":"c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783"} Sep 09 14:48:31.310875 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:31.310841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerStarted","Data":"b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c"} Sep 09 14:48:37.334557 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:37.334521 2578 generic.go:356] "Generic (PLEG): container finished" podID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerID="b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c" exitCode=0 Sep 09 14:48:37.334979 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:37.334562 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerDied","Data":"b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c"} Sep 09 14:48:38.342701 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:38.342661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerStarted","Data":"5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52"} Sep 09 14:48:38.343084 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:38.342743 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerStarted","Data":"cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b"} Sep 09 14:48:38.343084 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:38.342774 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerStarted","Data":"1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961"} Sep 09 14:48:38.365449 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:38.365381 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podStartSLOduration=11.365342164 podStartE2EDuration="11.365342164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:48:38.364940982 +0000 UTC m=+918.031747030" watchObservedRunningTime="2026-09-09 14:48:38.365342164 +0000 UTC m=+918.032148206" Sep 09 14:48:38.875719 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:38.875685 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:38.875898 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:38.875765 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:38.875898 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:38.875793 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:40.876309 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:40.876269 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:40.876748 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:40.876420 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:40.876748 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:40.876455 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:40.878284 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:40.878253 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 14:48:40.878406 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:40.878280 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 09 14:48:40.878406 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:40.878323 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 14:48:43.876839 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:43.876790 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 14:48:43.876839 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:43.876822 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 14:48:43.877431 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:43.876794 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Sep 09 14:48:46.876140 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:46.876084 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 14:48:46.877192 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:46.877163 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:46.878093 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:46.878071 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:46.882451 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:46.882431 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:47.391537 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:47.391507 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:49.876333 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:49.876291 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 09 14:48:52.876628 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:52.876593 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:48:52.877471 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:48:52.877449 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:49:57.054901 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.054868 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5"] Sep 09 14:49:57.055468 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.055104 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="registry" containerID="cri-o://1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961" gracePeriod=30 Sep 09 14:49:57.055468 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.055159 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="offline" containerID="cri-o://5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52" gracePeriod=30 Sep 09 14:49:57.055468 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.055162 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" containerID="cri-o://cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b" gracePeriod=30 Sep 09 14:49:57.076767 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.076737 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-9c9cs"] Sep 09 14:49:57.076937 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.076912 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-9c9cs" podUID="f720a1da-bd1c-4a18-ba81-f67da99fbc01" containerName="postgres" containerID="cri-o://84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64" gracePeriod=30 Sep 09 14:49:57.314336 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.314283 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:49:57.389041 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.388995 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 14:49:57.463580 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.463548 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f720a1da-bd1c-4a18-ba81-f67da99fbc01-postgresdata\") pod \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\" (UID: \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\") " Sep 09 14:49:57.463580 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.463591 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7rjk\" (UniqueName: \"kubernetes.io/projected/f720a1da-bd1c-4a18-ba81-f67da99fbc01-kube-api-access-s7rjk\") pod \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\" (UID: \"f720a1da-bd1c-4a18-ba81-f67da99fbc01\") " Sep 09 14:49:57.465810 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.465765 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f720a1da-bd1c-4a18-ba81-f67da99fbc01-kube-api-access-s7rjk" (OuterVolumeSpecName: "kube-api-access-s7rjk") pod "f720a1da-bd1c-4a18-ba81-f67da99fbc01" (UID: "f720a1da-bd1c-4a18-ba81-f67da99fbc01"). InnerVolumeSpecName "kube-api-access-s7rjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:49:57.496121 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.496085 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f720a1da-bd1c-4a18-ba81-f67da99fbc01-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "f720a1da-bd1c-4a18-ba81-f67da99fbc01" (UID: "f720a1da-bd1c-4a18-ba81-f67da99fbc01"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:49:57.564560 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.564482 2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f720a1da-bd1c-4a18-ba81-f67da99fbc01-postgresdata\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:49:57.564560 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.564509 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rjk\" (UniqueName: \"kubernetes.io/projected/f720a1da-bd1c-4a18-ba81-f67da99fbc01-kube-api-access-s7rjk\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:49:57.662567 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.662530 2578 generic.go:356] "Generic (PLEG): container finished" podID="f720a1da-bd1c-4a18-ba81-f67da99fbc01" containerID="84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64" exitCode=0 Sep 09 14:49:57.662762 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.662583 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-9c9cs" Sep 09 14:49:57.662762 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.662630 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-9c9cs" event={"ID":"f720a1da-bd1c-4a18-ba81-f67da99fbc01","Type":"ContainerDied","Data":"84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64"} Sep 09 14:49:57.662762 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.662688 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-9c9cs" event={"ID":"f720a1da-bd1c-4a18-ba81-f67da99fbc01","Type":"ContainerDied","Data":"125e601f658f1a23b839efa6c37f0f53c23fc2adc6c1047e8f17a0120c29213f"} Sep 09 14:49:57.662762 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.662709 2578 scope.go:117] "RemoveContainer" containerID="84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64" Sep 09 14:49:57.671336 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.671319 2578 scope.go:117] "RemoveContainer" containerID="84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64" Sep 09 14:49:57.671639 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:49:57.671616 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64\": container with ID starting with 84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64 not found: ID does not exist" containerID="84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64" Sep 09 14:49:57.671719 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.671647 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64"} err="failed to get container status \"84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64\": rpc error: code = NotFound desc = could not find container \"84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64\": container with ID starting with 84a8275afd4075a8d59e99f2d95b7136371e4f65f50c334347276b64afc99b64 not found: ID does not exist" Sep 09 14:49:57.682509 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.682478 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-9c9cs"] Sep 09 14:49:57.687589 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:57.687563 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-9c9cs"] Sep 09 14:49:58.670038 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:58.670004 2578 generic.go:356] "Generic (PLEG): container finished" podID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerID="cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b" exitCode=0 Sep 09 14:49:58.670415 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:58.670049 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerDied","Data":"cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b"} Sep 09 14:49:58.981547 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:49:58.981516 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f720a1da-bd1c-4a18-ba81-f67da99fbc01" path="/var/lib/kubelet/pods/f720a1da-bd1c-4a18-ba81-f67da99fbc01/volumes" Sep 09 14:50:07.389014 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:07.388968 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 14:50:17.389227 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:17.389181 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 14:50:17.389631 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:17.389305 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:50:27.389339 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.389242 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 09 14:50:27.701847 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.701819 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:50:27.778087 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.778051 2578 generic.go:356] "Generic (PLEG): container finished" podID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerID="5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52" exitCode=137 Sep 09 14:50:27.778087 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.778081 2578 generic.go:356] "Generic (PLEG): container finished" podID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerID="1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961" exitCode=137 Sep 09 14:50:27.778270 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.778106 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" Sep 09 14:50:27.778270 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.778139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerDied","Data":"5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52"} Sep 09 14:50:27.778270 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.778197 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerDied","Data":"1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961"} Sep 09 14:50:27.778270 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.778216 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5" event={"ID":"596f294d-8a20-4494-8dc4-e1ea20ef7153","Type":"ContainerDied","Data":"23833d2cd14912dc8da8c8031da5f7c354bf2660fbd79363a00b2b113906e37b"} Sep 09 14:50:27.778270 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.778236 2578 scope.go:117] "RemoveContainer" containerID="5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52" Sep 09 14:50:27.786109 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.786087 2578 scope.go:117] "RemoveContainer" containerID="cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b" Sep 09 14:50:27.793258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.793242 2578 scope.go:117] "RemoveContainer" containerID="1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961" Sep 09 14:50:27.797460 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.797440 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjb9p\" (UniqueName: \"kubernetes.io/projected/596f294d-8a20-4494-8dc4-e1ea20ef7153-kube-api-access-zjb9p\") pod \"596f294d-8a20-4494-8dc4-e1ea20ef7153\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " Sep 09 14:50:27.797546 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.797480 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-offline-tls\") pod \"596f294d-8a20-4494-8dc4-e1ea20ef7153\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " Sep 09 14:50:27.797546 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.797514 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-registry-tls\") pod \"596f294d-8a20-4494-8dc4-e1ea20ef7153\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " Sep 09 14:50:27.797664 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.797578 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-online-tls\") pod \"596f294d-8a20-4494-8dc4-e1ea20ef7153\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " Sep 09 14:50:27.797664 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.797612 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/596f294d-8a20-4494-8dc4-e1ea20ef7153-feast-data\") pod \"596f294d-8a20-4494-8dc4-e1ea20ef7153\" (UID: \"596f294d-8a20-4494-8dc4-e1ea20ef7153\") " Sep 09 14:50:27.799660 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.799632 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "596f294d-8a20-4494-8dc4-e1ea20ef7153" (UID: "596f294d-8a20-4494-8dc4-e1ea20ef7153"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:50:27.799754 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.799694 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596f294d-8a20-4494-8dc4-e1ea20ef7153-kube-api-access-zjb9p" (OuterVolumeSpecName: "kube-api-access-zjb9p") pod "596f294d-8a20-4494-8dc4-e1ea20ef7153" (UID: "596f294d-8a20-4494-8dc4-e1ea20ef7153"). InnerVolumeSpecName "kube-api-access-zjb9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:50:27.800049 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.800027 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "596f294d-8a20-4494-8dc4-e1ea20ef7153" (UID: "596f294d-8a20-4494-8dc4-e1ea20ef7153"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:50:27.800187 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.800166 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-online-tls" (OuterVolumeSpecName: "online-tls") pod "596f294d-8a20-4494-8dc4-e1ea20ef7153" (UID: "596f294d-8a20-4494-8dc4-e1ea20ef7153"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:50:27.801656 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.801636 2578 scope.go:117] "RemoveContainer" containerID="b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c" Sep 09 14:50:27.806532 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.806505 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596f294d-8a20-4494-8dc4-e1ea20ef7153-feast-data" (OuterVolumeSpecName: "feast-data") pod "596f294d-8a20-4494-8dc4-e1ea20ef7153" (UID: "596f294d-8a20-4494-8dc4-e1ea20ef7153"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:50:27.809495 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.809474 2578 scope.go:117] "RemoveContainer" containerID="c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783" Sep 09 14:50:27.816207 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.816189 2578 scope.go:117] "RemoveContainer" containerID="5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52" Sep 09 14:50:27.816466 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:50:27.816444 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52\": container with ID starting with 5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52 not found: ID does not exist" containerID="5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52" Sep 09 14:50:27.816538 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.816478 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52"} err="failed to get container status \"5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52\": rpc error: code = NotFound desc = could not find container \"5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52\": container with ID starting with 5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52 not found: ID does not exist" Sep 09 14:50:27.816538 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.816503 2578 scope.go:117] "RemoveContainer" containerID="cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b" Sep 09 14:50:27.816786 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:50:27.816769 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b\": container with ID starting with cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b not found: ID does not exist" containerID="cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b" Sep 09 14:50:27.816836 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.816791 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b"} err="failed to get container status \"cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b\": rpc error: code = NotFound desc = could not find container \"cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b\": container with ID starting with cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b not found: ID does not exist" Sep 09 14:50:27.816836 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.816806 2578 scope.go:117] "RemoveContainer" containerID="1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961" Sep 09 14:50:27.817057 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:50:27.817041 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961\": container with ID starting with 1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961 not found: ID does not exist" containerID="1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961" Sep 09 14:50:27.817101 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.817063 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961"} err="failed to get container status \"1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961\": rpc error: code = NotFound desc = could not find container \"1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961\": container with ID starting with 1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961 not found: ID does not exist" Sep 09 14:50:27.817101 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.817080 2578 scope.go:117] "RemoveContainer" containerID="b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c" Sep 09 14:50:27.817298 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:50:27.817278 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c\": container with ID starting with b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c not found: ID does not exist" containerID="b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c" Sep 09 14:50:27.817341 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.817293 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c"} err="failed to get container status \"b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c\": rpc error: code = NotFound desc = could not find container \"b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c\": container with ID starting with b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c not found: ID does not exist" Sep 09 14:50:27.817341 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.817306 2578 scope.go:117] "RemoveContainer" containerID="c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783" Sep 09 14:50:27.817662 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:50:27.817644 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783\": container with ID starting with c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783 not found: ID does not exist" containerID="c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783" Sep 09 14:50:27.817732 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.817670 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783"} err="failed to get container status \"c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783\": rpc error: code = NotFound desc = could not find container \"c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783\": container with ID starting with c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783 not found: ID does not exist" Sep 09 14:50:27.817732 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.817691 2578 scope.go:117] "RemoveContainer" containerID="5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52" Sep 09 14:50:27.817960 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.817934 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52"} err="failed to get container status \"5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52\": rpc error: code = NotFound desc = could not find container \"5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52\": container with ID starting with 5e2f2db942c710d1080d67a67028ac29e806136f83a125a3f7988a7abddeeb52 not found: ID does not exist" Sep 09 14:50:27.817960 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.817956 2578 scope.go:117] "RemoveContainer" containerID="cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b" Sep 09 14:50:27.818162 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.818147 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b"} err="failed to get container status \"cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b\": rpc error: code = NotFound desc = could not find container \"cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b\": container with ID starting with cdb9a99b62dbd7c3a659f4709582c8233bb1c5e624af064136fcc05e4f200a1b not found: ID does not exist" Sep 09 14:50:27.818162 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.818163 2578 scope.go:117] "RemoveContainer" containerID="1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961" Sep 09 14:50:27.818375 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.818340 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961"} err="failed to get container status \"1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961\": rpc error: code = NotFound desc = could not find container \"1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961\": container with ID starting with 1b057a53895e42fc9cadb9fb48cebd2e524620b025c5662e9fa2965cb946d961 not found: ID does not exist" Sep 09 14:50:27.818460 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.818376 2578 scope.go:117] "RemoveContainer" containerID="b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c" Sep 09 14:50:27.818569 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.818554 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c"} err="failed to get container status \"b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c\": rpc error: code = NotFound desc = could not find container \"b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c\": container with ID starting with b65130560322ff718587ba74878b46728d4c5b8ceea7efa31ce7e8099402f99c not found: ID does not exist" Sep 09 14:50:27.818609 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.818569 2578 scope.go:117] "RemoveContainer" containerID="c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783" Sep 09 14:50:27.818772 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.818753 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783"} err="failed to get container status \"c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783\": rpc error: code = NotFound desc = could not find container \"c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783\": container with ID starting with c3ae82050625dbcbd9ff6b09d1ea0fe8bf11a2e98d1073e50d55c58060196783 not found: ID does not exist" Sep 09 14:50:27.898425 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.898394 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjb9p\" (UniqueName: \"kubernetes.io/projected/596f294d-8a20-4494-8dc4-e1ea20ef7153-kube-api-access-zjb9p\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:50:27.898425 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.898422 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-offline-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:50:27.898425 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.898431 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-registry-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:50:27.898654 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.898439 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/596f294d-8a20-4494-8dc4-e1ea20ef7153-online-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:50:27.898654 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:27.898447 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/596f294d-8a20-4494-8dc4-e1ea20ef7153-feast-data\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:50:28.101283 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:28.101252 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5"] Sep 09 14:50:28.104777 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:28.104750 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-5b75c44fd-hhcl5"] Sep 09 14:50:28.981364 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:50:28.981321 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" path="/var/lib/kubelet/pods/596f294d-8a20-4494-8dc4-e1ea20ef7153/volumes" Sep 09 14:52:31.115599 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.115552 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d4zzq"] Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.115937 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="registry" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.115952 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="registry" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.115971 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="offline" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.115983 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="offline" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.115998 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="feast-apply" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116007 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="feast-apply" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116022 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116033 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116048 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f720a1da-bd1c-4a18-ba81-f67da99fbc01" containerName="postgres" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116057 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f720a1da-bd1c-4a18-ba81-f67da99fbc01" containerName="postgres" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116069 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="feast-init" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116078 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="feast-init" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116158 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="registry" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116169 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="offline" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116179 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="596f294d-8a20-4494-8dc4-e1ea20ef7153" containerName="online" Sep 09 14:52:31.116186 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.116189 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f720a1da-bd1c-4a18-ba81-f67da99fbc01" containerName="postgres" Sep 09 14:52:31.118905 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.118882 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" Sep 09 14:52:31.121776 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.121756 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 09 14:52:31.122165 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.122147 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 09 14:52:31.122586 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.122569 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-b5xqt\"" Sep 09 14:52:31.136231 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.136207 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d4zzq"] Sep 09 14:52:31.202314 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.202288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pmk\" (UniqueName: \"kubernetes.io/projected/166bed11-aeb3-4e6c-8adc-77904aee4228-kube-api-access-49pmk\") pod \"redis-69bbd798b6-d4zzq\" (UID: \"166bed11-aeb3-4e6c-8adc-77904aee4228\") " pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" Sep 09 14:52:31.302872 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.302841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pmk\" (UniqueName: \"kubernetes.io/projected/166bed11-aeb3-4e6c-8adc-77904aee4228-kube-api-access-49pmk\") pod \"redis-69bbd798b6-d4zzq\" (UID: \"166bed11-aeb3-4e6c-8adc-77904aee4228\") " pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" Sep 09 14:52:31.312198 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.312172 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pmk\" (UniqueName: \"kubernetes.io/projected/166bed11-aeb3-4e6c-8adc-77904aee4228-kube-api-access-49pmk\") pod \"redis-69bbd798b6-d4zzq\" (UID: \"166bed11-aeb3-4e6c-8adc-77904aee4228\") " pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" Sep 09 14:52:31.428656 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.428559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" Sep 09 14:52:31.705623 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:31.705542 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d4zzq"] Sep 09 14:52:31.709283 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:52:31.709254 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod166bed11_aeb3_4e6c_8adc_77904aee4228.slice/crio-b2361d3545bcdf8d43172eb5f17338c93e992e590d15755598aeb29925da692f WatchSource:0}: Error finding container b2361d3545bcdf8d43172eb5f17338c93e992e590d15755598aeb29925da692f: Status 404 returned error can't find the container with id b2361d3545bcdf8d43172eb5f17338c93e992e590d15755598aeb29925da692f Sep 09 14:52:32.215027 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:32.214990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" event={"ID":"166bed11-aeb3-4e6c-8adc-77904aee4228","Type":"ContainerStarted","Data":"b2361d3545bcdf8d43172eb5f17338c93e992e590d15755598aeb29925da692f"} Sep 09 14:52:33.220395 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:33.220342 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" event={"ID":"166bed11-aeb3-4e6c-8adc-77904aee4228","Type":"ContainerStarted","Data":"10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2"} Sep 09 14:52:33.235516 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:33.235470 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" podStartSLOduration=0.853749596 podStartE2EDuration="2.235455452s" totalImagesPullingTime="1.381705856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:52:33.234680699 +0000 UTC m=+1152.901486748" watchObservedRunningTime="2026-09-09 14:52:33.235455452 +0000 UTC m=+1152.902261499" Sep 09 14:52:36.543067 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.543035 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-cfc4b"] Sep 09 14:52:36.546528 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.546503 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:52:36.548903 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.548880 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 09 14:52:36.554798 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.554774 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-cfc4b"] Sep 09 14:52:36.647572 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.647534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m96bs\" (UniqueName: \"kubernetes.io/projected/ee087239-2846-4685-86aa-cd60e9f62cc9-kube-api-access-m96bs\") pod \"postgres-864846d4f5-cfc4b\" (UID: \"ee087239-2846-4685-86aa-cd60e9f62cc9\") " pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:52:36.647769 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.647598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee087239-2846-4685-86aa-cd60e9f62cc9-postgresdata\") pod \"postgres-864846d4f5-cfc4b\" (UID: \"ee087239-2846-4685-86aa-cd60e9f62cc9\") " pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:52:36.748235 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.748197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee087239-2846-4685-86aa-cd60e9f62cc9-postgresdata\") pod \"postgres-864846d4f5-cfc4b\" (UID: \"ee087239-2846-4685-86aa-cd60e9f62cc9\") " pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:52:36.748431 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.748264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m96bs\" (UniqueName: \"kubernetes.io/projected/ee087239-2846-4685-86aa-cd60e9f62cc9-kube-api-access-m96bs\") pod \"postgres-864846d4f5-cfc4b\" (UID: \"ee087239-2846-4685-86aa-cd60e9f62cc9\") " pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:52:36.748666 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.748641 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee087239-2846-4685-86aa-cd60e9f62cc9-postgresdata\") pod \"postgres-864846d4f5-cfc4b\" (UID: \"ee087239-2846-4685-86aa-cd60e9f62cc9\") " pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:52:36.756157 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.756138 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m96bs\" (UniqueName: \"kubernetes.io/projected/ee087239-2846-4685-86aa-cd60e9f62cc9-kube-api-access-m96bs\") pod \"postgres-864846d4f5-cfc4b\" (UID: \"ee087239-2846-4685-86aa-cd60e9f62cc9\") " pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:52:36.857275 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:36.857203 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:52:37.122744 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:37.122667 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-cfc4b"] Sep 09 14:52:37.125857 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:52:37.125834 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee087239_2846_4685_86aa_cd60e9f62cc9.slice/crio-85e4dce2899542136b7bc945bb6b3b7584c5c4da5b77977b4cb1671493094c2c WatchSource:0}: Error finding container 85e4dce2899542136b7bc945bb6b3b7584c5c4da5b77977b4cb1671493094c2c: Status 404 returned error can't find the container with id 85e4dce2899542136b7bc945bb6b3b7584c5c4da5b77977b4cb1671493094c2c Sep 09 14:52:37.237122 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:37.237084 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" event={"ID":"ee087239-2846-4685-86aa-cd60e9f62cc9","Type":"ContainerStarted","Data":"b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c"} Sep 09 14:52:37.237122 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:37.237125 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" event={"ID":"ee087239-2846-4685-86aa-cd60e9f62cc9","Type":"ContainerStarted","Data":"85e4dce2899542136b7bc945bb6b3b7584c5c4da5b77977b4cb1671493094c2c"} Sep 09 14:52:37.251089 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:37.251024 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" podStartSLOduration=1.251002787 podStartE2EDuration="1.251002787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:52:37.250602973 +0000 UTC m=+1156.917409021" watchObservedRunningTime="2026-09-09 14:52:37.251002787 +0000 UTC m=+1156.917808833" Sep 09 14:52:42.124361 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.124309 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6"] Sep 09 14:52:42.127627 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.127610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.130258 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.130238 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 09 14:52:42.130496 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.130482 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-xjjgj\"" Sep 09 14:52:42.130603 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.130490 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 09 14:52:42.130713 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.130699 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 09 14:52:42.141631 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.141612 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6"] Sep 09 14:52:42.294205 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.294175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-online-tls\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.294375 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.294229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-registry-tls\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.294375 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.294284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb9ff8-0669-421d-904b-fd48e4abd21e-feast-data\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.294375 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.294357 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8zm\" (UniqueName: \"kubernetes.io/projected/cceb9ff8-0669-421d-904b-fd48e4abd21e-kube-api-access-bn8zm\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.395479 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.395401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-online-tls\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.395479 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.395456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-registry-tls\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.395479 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.395473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb9ff8-0669-421d-904b-fd48e4abd21e-feast-data\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.395680 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.395647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8zm\" (UniqueName: \"kubernetes.io/projected/cceb9ff8-0669-421d-904b-fd48e4abd21e-kube-api-access-bn8zm\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.395904 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.395870 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb9ff8-0669-421d-904b-fd48e4abd21e-feast-data\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.398018 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.397993 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-registry-tls\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.398125 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.398033 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-online-tls\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.403127 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.403103 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8zm\" (UniqueName: \"kubernetes.io/projected/cceb9ff8-0669-421d-904b-fd48e4abd21e-kube-api-access-bn8zm\") pod \"feast-credit-scoring-788cb688d9-bntn6\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.436297 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.436265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:42.708659 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:42.708634 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6"] Sep 09 14:52:42.711173 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:52:42.711149 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcceb9ff8_0669_421d_904b_fd48e4abd21e.slice/crio-d415cd8cca023694dd2e8916bbf8fcfa7245106b51ecd143d2a84ce3538e8b68 WatchSource:0}: Error finding container d415cd8cca023694dd2e8916bbf8fcfa7245106b51ecd143d2a84ce3538e8b68: Status 404 returned error can't find the container with id d415cd8cca023694dd2e8916bbf8fcfa7245106b51ecd143d2a84ce3538e8b68 Sep 09 14:52:43.262016 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:43.261970 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerStarted","Data":"c730389bee2c250b80243696422e9e0f3bccfeb2a0f05e44e145e73d353bdf62"} Sep 09 14:52:43.262570 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:43.262036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerStarted","Data":"d415cd8cca023694dd2e8916bbf8fcfa7245106b51ecd143d2a84ce3538e8b68"} Sep 09 14:52:44.267579 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:44.267545 2578 generic.go:356] "Generic (PLEG): container finished" podID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerID="c730389bee2c250b80243696422e9e0f3bccfeb2a0f05e44e145e73d353bdf62" exitCode=0 Sep 09 14:52:44.267933 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:44.267626 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerDied","Data":"c730389bee2c250b80243696422e9e0f3bccfeb2a0f05e44e145e73d353bdf62"} Sep 09 14:52:45.272696 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:45.272661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerStarted","Data":"ea82f71097fe968bc728e3c144fdd49f725db5fbf29277a144f19c695bad4ab5"} Sep 09 14:52:51.299236 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:51.299203 2578 generic.go:356] "Generic (PLEG): container finished" podID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerID="ea82f71097fe968bc728e3c144fdd49f725db5fbf29277a144f19c695bad4ab5" exitCode=0 Sep 09 14:52:51.299630 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:51.299248 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerDied","Data":"ea82f71097fe968bc728e3c144fdd49f725db5fbf29277a144f19c695bad4ab5"} Sep 09 14:52:52.306241 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:52.306206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerStarted","Data":"4efb992e8607b7f90db76a8df40ad75d4058e1fe7da28f9d647bae1a9a078f3f"} Sep 09 14:52:52.306732 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:52.306298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerStarted","Data":"0b3ade39a30344d92b7ce0c62e84b6fd987186ec55e52bd88659e0ebf7008e5d"} Sep 09 14:52:52.327958 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:52.327913 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podStartSLOduration=10.327899203 podStartE2EDuration="10.327899203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:52:52.32671449 +0000 UTC m=+1171.993520537" watchObservedRunningTime="2026-09-09 14:52:52.327899203 +0000 UTC m=+1171.994705249" Sep 09 14:52:52.436455 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:52.436417 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:52.436591 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:52.436489 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:54.437419 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:54.437382 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:54.437803 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:54.437447 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:52:54.439303 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:54.439268 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 09 14:52:54.439437 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:54.439299 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 14:52:57.437327 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:57.437265 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 09 14:52:57.437809 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:52:57.437280 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 14:53:00.437179 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:00.437127 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Sep 09 14:53:00.443167 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:00.443142 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:53:00.446460 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:00.446437 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:53:03.437590 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:03.437559 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:53:03.438337 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:03.438317 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:53:08.001409 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.001368 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb"] Sep 09 14:53:08.008047 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.008017 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.010247 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.010219 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 09 14:53:08.010420 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.010398 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-cwb8z\"" Sep 09 14:53:08.010499 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.010480 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 09 14:53:08.010560 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.010405 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 09 14:53:08.016124 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.016102 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb"] Sep 09 14:53:08.096032 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.095998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-registry-tls\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.096032 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.096039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-online-tls\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.096229 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.096054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-feast-data\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.096229 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.096143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657t8\" (UniqueName: \"kubernetes.io/projected/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-kube-api-access-657t8\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.196851 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.196815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-online-tls\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.196851 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.196850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-feast-data\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.197065 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.196882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657t8\" (UniqueName: \"kubernetes.io/projected/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-kube-api-access-657t8\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.197065 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.196914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-registry-tls\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.197206 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.197186 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-feast-data\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.199407 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.199380 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-online-tls\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.199407 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.199401 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-registry-tls\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.206943 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.206923 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-657t8\" (UniqueName: \"kubernetes.io/projected/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-kube-api-access-657t8\") pod \"feast-driver-ranking-567f5596cd-4mtwb\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.317129 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.317044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:08.594086 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:08.594060 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb"] Sep 09 14:53:08.596445 ip-10-0-130-173 kubenswrapper[2578]: W0909 14:53:08.596416 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a46bae8_7e65_4dd2_b66d_e2fce80e5487.slice/crio-0ec55f929384aa15117700c20c05ba3f0e1ae8c46dbe0f7613913fcd422d25aa WatchSource:0}: Error finding container 0ec55f929384aa15117700c20c05ba3f0e1ae8c46dbe0f7613913fcd422d25aa: Status 404 returned error can't find the container with id 0ec55f929384aa15117700c20c05ba3f0e1ae8c46dbe0f7613913fcd422d25aa Sep 09 14:53:09.386167 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:09.386125 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerStarted","Data":"94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792"} Sep 09 14:53:09.386647 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:09.386193 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerStarted","Data":"0ec55f929384aa15117700c20c05ba3f0e1ae8c46dbe0f7613913fcd422d25aa"} Sep 09 14:53:13.404024 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:13.403988 2578 generic.go:356] "Generic (PLEG): container finished" podID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerID="94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792" exitCode=0 Sep 09 14:53:13.404392 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:13.404060 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerDied","Data":"94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792"} Sep 09 14:53:14.409980 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:14.409946 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerStarted","Data":"bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b"} Sep 09 14:53:20.436202 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:20.436166 2578 generic.go:356] "Generic (PLEG): container finished" podID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerID="bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b" exitCode=0 Sep 09 14:53:20.436692 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:20.436225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerDied","Data":"bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b"} Sep 09 14:53:20.867625 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:20.867599 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:53:20.869439 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:20.869418 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:53:21.443995 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:21.443960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerStarted","Data":"641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5"} Sep 09 14:53:21.444408 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:21.444019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerStarted","Data":"d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6"} Sep 09 14:53:21.477325 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:21.477272 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podStartSLOduration=14.477256314 podStartE2EDuration="14.477256314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 14:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 14:53:21.474887921 +0000 UTC m=+1201.141693970" watchObservedRunningTime="2026-09-09 14:53:21.477256314 +0000 UTC m=+1201.144062357" Sep 09 14:53:23.317411 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:23.317369 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:23.317875 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:23.317453 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:23.319104 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:23.319068 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 09 14:53:23.319253 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:23.319115 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 14:53:26.317616 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:26.317564 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 09 14:53:26.317616 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:26.317601 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 14:53:28.317176 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:28.317132 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:28.317678 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:28.317203 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:29.317973 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:29.317924 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Sep 09 14:53:29.323906 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:29.323876 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:29.327438 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:29.327412 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:32.318604 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:32.318572 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:53:32.319319 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:53:32.319300 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:54:50.591291 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.591256 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc"] Sep 09 14:54:50.591776 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.591478 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" podUID="8140b4f0-6fff-4f41-acb4-4cd03c913cdf" containerName="manager" containerID="cri-o://2f7a156a77a3ef14c5097a135e54d7ea2872bae7c721688ff546901e3983cc03" gracePeriod=10 Sep 09 14:54:50.744791 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.744752 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" podUID="8140b4f0-6fff-4f41-acb4-4cd03c913cdf" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.23:8081/readyz\": dial tcp 10.134.0.23:8081: connect: connection refused" Sep 09 14:54:50.806682 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.806649 2578 generic.go:356] "Generic (PLEG): container finished" podID="8140b4f0-6fff-4f41-acb4-4cd03c913cdf" containerID="2f7a156a77a3ef14c5097a135e54d7ea2872bae7c721688ff546901e3983cc03" exitCode=0 Sep 09 14:54:50.806848 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.806732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" event={"ID":"8140b4f0-6fff-4f41-acb4-4cd03c913cdf","Type":"ContainerDied","Data":"2f7a156a77a3ef14c5097a135e54d7ea2872bae7c721688ff546901e3983cc03"} Sep 09 14:54:50.850942 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.850888 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" Sep 09 14:54:50.870065 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.870025 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6"] Sep 09 14:54:50.870283 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.870255 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="registry" containerID="cri-o://0b3ade39a30344d92b7ce0c62e84b6fd987186ec55e52bd88659e0ebf7008e5d" gracePeriod=30 Sep 09 14:54:50.870397 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.870301 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="online" containerID="cri-o://4efb992e8607b7f90db76a8df40ad75d4058e1fe7da28f9d647bae1a9a078f3f" gracePeriod=30 Sep 09 14:54:50.879439 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.879401 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb"] Sep 09 14:54:50.879743 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.879693 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="registry" containerID="cri-o://d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6" gracePeriod=30 Sep 09 14:54:50.879929 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:50.879906 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="online" containerID="cri-o://641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5" gracePeriod=30 Sep 09 14:54:51.016412 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.016382 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvv96\" (UniqueName: \"kubernetes.io/projected/8140b4f0-6fff-4f41-acb4-4cd03c913cdf-kube-api-access-xvv96\") pod \"8140b4f0-6fff-4f41-acb4-4cd03c913cdf\" (UID: \"8140b4f0-6fff-4f41-acb4-4cd03c913cdf\") " Sep 09 14:54:51.018488 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.018460 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8140b4f0-6fff-4f41-acb4-4cd03c913cdf-kube-api-access-xvv96" (OuterVolumeSpecName: "kube-api-access-xvv96") pod "8140b4f0-6fff-4f41-acb4-4cd03c913cdf" (UID: "8140b4f0-6fff-4f41-acb4-4cd03c913cdf"). InnerVolumeSpecName "kube-api-access-xvv96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:54:51.117339 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.117256 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvv96\" (UniqueName: \"kubernetes.io/projected/8140b4f0-6fff-4f41-acb4-4cd03c913cdf-kube-api-access-xvv96\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:54:51.811393 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.811338 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" Sep 09 14:54:51.811393 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.811339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc" event={"ID":"8140b4f0-6fff-4f41-acb4-4cd03c913cdf","Type":"ContainerDied","Data":"6958db78375510e61cae23fe8311aa141005da3e2af939f446ec722db0b88d05"} Sep 09 14:54:51.811891 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.811431 2578 scope.go:117] "RemoveContainer" containerID="2f7a156a77a3ef14c5097a135e54d7ea2872bae7c721688ff546901e3983cc03" Sep 09 14:54:51.814159 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.814134 2578 generic.go:356] "Generic (PLEG): container finished" podID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerID="4efb992e8607b7f90db76a8df40ad75d4058e1fe7da28f9d647bae1a9a078f3f" exitCode=0 Sep 09 14:54:51.814262 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.814210 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerDied","Data":"4efb992e8607b7f90db76a8df40ad75d4058e1fe7da28f9d647bae1a9a078f3f"} Sep 09 14:54:51.816910 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.816890 2578 generic.go:356] "Generic (PLEG): container finished" podID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerID="641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5" exitCode=0 Sep 09 14:54:51.817014 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.816952 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerDied","Data":"641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5"} Sep 09 14:54:51.831992 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.831971 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc"] Sep 09 14:54:51.836613 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:51.836593 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-9c5b6dd6c-z4wcc"] Sep 09 14:54:52.981513 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:52.981482 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8140b4f0-6fff-4f41-acb4-4cd03c913cdf" path="/var/lib/kubelet/pods/8140b4f0-6fff-4f41-acb4-4cd03c913cdf/volumes" Sep 09 14:54:53.179508 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.179476 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-cfc4b"] Sep 09 14:54:53.179670 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.179641 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" podUID="ee087239-2846-4685-86aa-cd60e9f62cc9" containerName="postgres" containerID="cri-o://b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c" gracePeriod=30 Sep 09 14:54:53.184291 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.184266 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d4zzq"] Sep 09 14:54:53.184459 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.184439 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" podUID="166bed11-aeb3-4e6c-8adc-77904aee4228" containerName="redis" containerID="cri-o://10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2" gracePeriod=30 Sep 09 14:54:53.457319 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.457273 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:54:53.459770 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.459752 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" Sep 09 14:54:53.638758 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.638669 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m96bs\" (UniqueName: \"kubernetes.io/projected/ee087239-2846-4685-86aa-cd60e9f62cc9-kube-api-access-m96bs\") pod \"ee087239-2846-4685-86aa-cd60e9f62cc9\" (UID: \"ee087239-2846-4685-86aa-cd60e9f62cc9\") " Sep 09 14:54:53.638758 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.638730 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee087239-2846-4685-86aa-cd60e9f62cc9-postgresdata\") pod \"ee087239-2846-4685-86aa-cd60e9f62cc9\" (UID: \"ee087239-2846-4685-86aa-cd60e9f62cc9\") " Sep 09 14:54:53.638937 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.638763 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49pmk\" (UniqueName: \"kubernetes.io/projected/166bed11-aeb3-4e6c-8adc-77904aee4228-kube-api-access-49pmk\") pod \"166bed11-aeb3-4e6c-8adc-77904aee4228\" (UID: \"166bed11-aeb3-4e6c-8adc-77904aee4228\") " Sep 09 14:54:53.641010 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.640978 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee087239-2846-4685-86aa-cd60e9f62cc9-kube-api-access-m96bs" (OuterVolumeSpecName: "kube-api-access-m96bs") pod "ee087239-2846-4685-86aa-cd60e9f62cc9" (UID: "ee087239-2846-4685-86aa-cd60e9f62cc9"). InnerVolumeSpecName "kube-api-access-m96bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:54:53.641010 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.640976 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166bed11-aeb3-4e6c-8adc-77904aee4228-kube-api-access-49pmk" (OuterVolumeSpecName: "kube-api-access-49pmk") pod "166bed11-aeb3-4e6c-8adc-77904aee4228" (UID: "166bed11-aeb3-4e6c-8adc-77904aee4228"). InnerVolumeSpecName "kube-api-access-49pmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:54:53.668648 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.668608 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee087239-2846-4685-86aa-cd60e9f62cc9-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "ee087239-2846-4685-86aa-cd60e9f62cc9" (UID: "ee087239-2846-4685-86aa-cd60e9f62cc9"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:54:53.739708 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.739679 2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee087239-2846-4685-86aa-cd60e9f62cc9-postgresdata\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:54:53.739708 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.739704 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49pmk\" (UniqueName: \"kubernetes.io/projected/166bed11-aeb3-4e6c-8adc-77904aee4228-kube-api-access-49pmk\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:54:53.739708 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.739714 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m96bs\" (UniqueName: \"kubernetes.io/projected/ee087239-2846-4685-86aa-cd60e9f62cc9-kube-api-access-m96bs\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:54:53.826181 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.826147 2578 generic.go:356] "Generic (PLEG): container finished" podID="166bed11-aeb3-4e6c-8adc-77904aee4228" containerID="10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2" exitCode=0 Sep 09 14:54:53.826340 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.826187 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" Sep 09 14:54:53.826340 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.826225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" event={"ID":"166bed11-aeb3-4e6c-8adc-77904aee4228","Type":"ContainerDied","Data":"10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2"} Sep 09 14:54:53.826340 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.826262 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-d4zzq" event={"ID":"166bed11-aeb3-4e6c-8adc-77904aee4228","Type":"ContainerDied","Data":"b2361d3545bcdf8d43172eb5f17338c93e992e590d15755598aeb29925da692f"} Sep 09 14:54:53.826340 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.826279 2578 scope.go:117] "RemoveContainer" containerID="10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2" Sep 09 14:54:53.827558 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.827539 2578 generic.go:356] "Generic (PLEG): container finished" podID="ee087239-2846-4685-86aa-cd60e9f62cc9" containerID="b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c" exitCode=0 Sep 09 14:54:53.827669 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.827609 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" Sep 09 14:54:53.827735 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.827614 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" event={"ID":"ee087239-2846-4685-86aa-cd60e9f62cc9","Type":"ContainerDied","Data":"b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c"} Sep 09 14:54:53.827735 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.827715 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-cfc4b" event={"ID":"ee087239-2846-4685-86aa-cd60e9f62cc9","Type":"ContainerDied","Data":"85e4dce2899542136b7bc945bb6b3b7584c5c4da5b77977b4cb1671493094c2c"} Sep 09 14:54:53.835818 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.835799 2578 scope.go:117] "RemoveContainer" containerID="10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2" Sep 09 14:54:53.836107 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:54:53.836084 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2\": container with ID starting with 10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2 not found: ID does not exist" containerID="10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2" Sep 09 14:54:53.836152 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.836120 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2"} err="failed to get container status \"10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2\": rpc error: code = NotFound desc = could not find container \"10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2\": container with ID starting with 10afabb9970b90ca09d8e995e68f505dddf64ec031c09020562401730f84c6e2 not found: ID does not exist" Sep 09 14:54:53.836152 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.836142 2578 scope.go:117] "RemoveContainer" containerID="b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c" Sep 09 14:54:53.843692 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.843678 2578 scope.go:117] "RemoveContainer" containerID="b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c" Sep 09 14:54:53.843941 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:54:53.843919 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c\": container with ID starting with b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c not found: ID does not exist" containerID="b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c" Sep 09 14:54:53.844017 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.843945 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c"} err="failed to get container status \"b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c\": rpc error: code = NotFound desc = could not find container \"b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c\": container with ID starting with b14b670791067dbdb158c2a6394e1925147dbcf4ae4678c7d15acb6752eced4c not found: ID does not exist" Sep 09 14:54:53.850904 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.850884 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-cfc4b"] Sep 09 14:54:53.854928 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.854909 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-cfc4b"] Sep 09 14:54:53.864274 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.864255 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d4zzq"] Sep 09 14:54:53.869913 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:53.869895 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d4zzq"] Sep 09 14:54:54.985091 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:54.982973 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="166bed11-aeb3-4e6c-8adc-77904aee4228" path="/var/lib/kubelet/pods/166bed11-aeb3-4e6c-8adc-77904aee4228/volumes" Sep 09 14:54:54.985091 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:54.983509 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee087239-2846-4685-86aa-cd60e9f62cc9" path="/var/lib/kubelet/pods/ee087239-2846-4685-86aa-cd60e9f62cc9/volumes" Sep 09 14:54:59.325012 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:54:59.324967 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 14:55:00.444604 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:00.444560 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 14:55:09.325444 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:09.325403 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 14:55:10.443751 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:10.443712 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 14:55:19.324926 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:19.324887 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Sep 09 14:55:19.325409 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:19.324993 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:55:20.444014 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:20.443973 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Sep 09 14:55:20.444409 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:20.444080 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:55:20.934268 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:20.932329 2578 generic.go:356] "Generic (PLEG): container finished" podID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerID="0b3ade39a30344d92b7ce0c62e84b6fd987186ec55e52bd88659e0ebf7008e5d" exitCode=137 Sep 09 14:55:20.934268 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:20.932375 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerDied","Data":"0b3ade39a30344d92b7ce0c62e84b6fd987186ec55e52bd88659e0ebf7008e5d"} Sep 09 14:55:21.552747 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.552722 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:55:21.555382 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.555363 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:55:21.633518 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.633489 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-online-tls\") pod \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " Sep 09 14:55:21.633707 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.633534 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-feast-data\") pod \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " Sep 09 14:55:21.633707 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.633550 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-registry-tls\") pod \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " Sep 09 14:55:21.633707 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.633567 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-online-tls\") pod \"cceb9ff8-0669-421d-904b-fd48e4abd21e\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " Sep 09 14:55:21.633707 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.633584 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-registry-tls\") pod \"cceb9ff8-0669-421d-904b-fd48e4abd21e\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " Sep 09 14:55:21.633707 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.633683 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn8zm\" (UniqueName: \"kubernetes.io/projected/cceb9ff8-0669-421d-904b-fd48e4abd21e-kube-api-access-bn8zm\") pod \"cceb9ff8-0669-421d-904b-fd48e4abd21e\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " Sep 09 14:55:21.633986 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.633739 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb9ff8-0669-421d-904b-fd48e4abd21e-feast-data\") pod \"cceb9ff8-0669-421d-904b-fd48e4abd21e\" (UID: \"cceb9ff8-0669-421d-904b-fd48e4abd21e\") " Sep 09 14:55:21.633986 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.633771 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-657t8\" (UniqueName: \"kubernetes.io/projected/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-kube-api-access-657t8\") pod \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\" (UID: \"8a46bae8-7e65-4dd2-b66d-e2fce80e5487\") " Sep 09 14:55:21.634086 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.634061 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-feast-data" (OuterVolumeSpecName: "feast-data") pod "8a46bae8-7e65-4dd2-b66d-e2fce80e5487" (UID: "8a46bae8-7e65-4dd2-b66d-e2fce80e5487"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:55:21.635988 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.635957 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cceb9ff8-0669-421d-904b-fd48e4abd21e" (UID: "cceb9ff8-0669-421d-904b-fd48e4abd21e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:55:21.636196 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.636164 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-online-tls" (OuterVolumeSpecName: "online-tls") pod "cceb9ff8-0669-421d-904b-fd48e4abd21e" (UID: "cceb9ff8-0669-421d-904b-fd48e4abd21e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:55:21.636268 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.636189 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-online-tls" (OuterVolumeSpecName: "online-tls") pod "8a46bae8-7e65-4dd2-b66d-e2fce80e5487" (UID: "8a46bae8-7e65-4dd2-b66d-e2fce80e5487"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:55:21.636268 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.636169 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8a46bae8-7e65-4dd2-b66d-e2fce80e5487" (UID: "8a46bae8-7e65-4dd2-b66d-e2fce80e5487"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 14:55:21.636268 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.636216 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cceb9ff8-0669-421d-904b-fd48e4abd21e-kube-api-access-bn8zm" (OuterVolumeSpecName: "kube-api-access-bn8zm") pod "cceb9ff8-0669-421d-904b-fd48e4abd21e" (UID: "cceb9ff8-0669-421d-904b-fd48e4abd21e"). InnerVolumeSpecName "kube-api-access-bn8zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:55:21.636268 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.636260 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-kube-api-access-657t8" (OuterVolumeSpecName: "kube-api-access-657t8") pod "8a46bae8-7e65-4dd2-b66d-e2fce80e5487" (UID: "8a46bae8-7e65-4dd2-b66d-e2fce80e5487"). InnerVolumeSpecName "kube-api-access-657t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 14:55:21.643532 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.643489 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cceb9ff8-0669-421d-904b-fd48e4abd21e-feast-data" (OuterVolumeSpecName: "feast-data") pod "cceb9ff8-0669-421d-904b-fd48e4abd21e" (UID: "cceb9ff8-0669-421d-904b-fd48e4abd21e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 14:55:21.734702 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.734674 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-online-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:55:21.734702 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.734696 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-feast-data\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:55:21.734702 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.734705 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-registry-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:55:21.734876 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.734713 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-online-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:55:21.734876 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.734720 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cceb9ff8-0669-421d-904b-fd48e4abd21e-registry-tls\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:55:21.734876 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.734733 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn8zm\" (UniqueName: \"kubernetes.io/projected/cceb9ff8-0669-421d-904b-fd48e4abd21e-kube-api-access-bn8zm\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:55:21.734876 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.734742 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cceb9ff8-0669-421d-904b-fd48e4abd21e-feast-data\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:55:21.734876 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.734751 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657t8\" (UniqueName: \"kubernetes.io/projected/8a46bae8-7e65-4dd2-b66d-e2fce80e5487-kube-api-access-657t8\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 14:55:21.938548 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.938473 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" Sep 09 14:55:21.938709 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.938466 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6" event={"ID":"cceb9ff8-0669-421d-904b-fd48e4abd21e","Type":"ContainerDied","Data":"d415cd8cca023694dd2e8916bbf8fcfa7245106b51ecd143d2a84ce3538e8b68"} Sep 09 14:55:21.938709 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.938653 2578 scope.go:117] "RemoveContainer" containerID="4efb992e8607b7f90db76a8df40ad75d4058e1fe7da28f9d647bae1a9a078f3f" Sep 09 14:55:21.940840 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.940816 2578 generic.go:356] "Generic (PLEG): container finished" podID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerID="d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6" exitCode=137 Sep 09 14:55:21.940943 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.940882 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" Sep 09 14:55:21.940943 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.940891 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerDied","Data":"d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6"} Sep 09 14:55:21.940943 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.940929 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb" event={"ID":"8a46bae8-7e65-4dd2-b66d-e2fce80e5487","Type":"ContainerDied","Data":"0ec55f929384aa15117700c20c05ba3f0e1ae8c46dbe0f7613913fcd422d25aa"} Sep 09 14:55:21.947254 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.947233 2578 scope.go:117] "RemoveContainer" containerID="0b3ade39a30344d92b7ce0c62e84b6fd987186ec55e52bd88659e0ebf7008e5d" Sep 09 14:55:21.955281 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.955267 2578 scope.go:117] "RemoveContainer" containerID="ea82f71097fe968bc728e3c144fdd49f725db5fbf29277a144f19c695bad4ab5" Sep 09 14:55:21.961922 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.961901 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6"] Sep 09 14:55:21.962212 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.962198 2578 scope.go:117] "RemoveContainer" containerID="c730389bee2c250b80243696422e9e0f3bccfeb2a0f05e44e145e73d353bdf62" Sep 09 14:55:21.966883 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.966861 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-788cb688d9-bntn6"] Sep 09 14:55:21.969457 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.969439 2578 scope.go:117] "RemoveContainer" containerID="641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5" Sep 09 14:55:21.976480 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.976408 2578 scope.go:117] "RemoveContainer" containerID="d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6" Sep 09 14:55:21.976751 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.976716 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb"] Sep 09 14:55:21.980955 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.980934 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-567f5596cd-4mtwb"] Sep 09 14:55:21.983736 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.983723 2578 scope.go:117] "RemoveContainer" containerID="bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b" Sep 09 14:55:21.990118 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.990102 2578 scope.go:117] "RemoveContainer" containerID="94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792" Sep 09 14:55:21.996669 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.996651 2578 scope.go:117] "RemoveContainer" containerID="641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5" Sep 09 14:55:21.996897 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:55:21.996879 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5\": container with ID starting with 641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5 not found: ID does not exist" containerID="641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5" Sep 09 14:55:21.996939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.996905 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5"} err="failed to get container status \"641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5\": rpc error: code = NotFound desc = could not find container \"641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5\": container with ID starting with 641149ca8ff300be737ec0ffd61b63bf5189d34e6ad2dd06f1900831e78917c5 not found: ID does not exist" Sep 09 14:55:21.996939 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.996922 2578 scope.go:117] "RemoveContainer" containerID="d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6" Sep 09 14:55:21.997198 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:55:21.997180 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6\": container with ID starting with d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6 not found: ID does not exist" containerID="d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6" Sep 09 14:55:21.997239 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.997215 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6"} err="failed to get container status \"d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6\": rpc error: code = NotFound desc = could not find container \"d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6\": container with ID starting with d0cb99e271dbbb7762370025ef25a9a25b6c4c71062e477f3e257e95ab238be6 not found: ID does not exist" Sep 09 14:55:21.997239 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.997231 2578 scope.go:117] "RemoveContainer" containerID="bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b" Sep 09 14:55:21.997462 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:55:21.997438 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b\": container with ID starting with bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b not found: ID does not exist" containerID="bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b" Sep 09 14:55:21.997527 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.997468 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b"} err="failed to get container status \"bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b\": rpc error: code = NotFound desc = could not find container \"bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b\": container with ID starting with bb9c0d54a4202dbbfd690eedfb07e2e8c997e2053481358eb77c05d3b7abf92b not found: ID does not exist" Sep 09 14:55:21.997527 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.997486 2578 scope.go:117] "RemoveContainer" containerID="94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792" Sep 09 14:55:21.997730 ip-10-0-130-173 kubenswrapper[2578]: E0909 14:55:21.997717 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792\": container with ID starting with 94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792 not found: ID does not exist" containerID="94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792" Sep 09 14:55:21.997773 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:21.997733 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792"} err="failed to get container status \"94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792\": rpc error: code = NotFound desc = could not find container \"94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792\": container with ID starting with 94ccd67485d1e5899f98564b8143439c5b98b399d9c03f4282e7becb59c5e792 not found: ID does not exist" Sep 09 14:55:22.980688 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:22.980656 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" path="/var/lib/kubelet/pods/8a46bae8-7e65-4dd2-b66d-e2fce80e5487/volumes" Sep 09 14:55:22.981147 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:55:22.981133 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" path="/var/lib/kubelet/pods/cceb9ff8-0669-421d-904b-fd48e4abd21e/volumes" Sep 09 14:58:20.892415 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:58:20.892382 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 14:58:20.894127 ip-10-0-130-173 kubenswrapper[2578]: I0909 14:58:20.894102 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 15:03:20.913362 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:03:20.913312 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 15:03:20.915194 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:03:20.915170 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 15:06:48.056497 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056459 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rdv72/must-gather-2ch79"] Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056706 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="online" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056717 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="online" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056728 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee087239-2846-4685-86aa-cd60e9f62cc9" containerName="postgres" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056733 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee087239-2846-4685-86aa-cd60e9f62cc9" containerName="postgres" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056740 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="feast-apply" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056745 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="feast-apply" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056755 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="online" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056760 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="online" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056767 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8140b4f0-6fff-4f41-acb4-4cd03c913cdf" containerName="manager" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056772 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8140b4f0-6fff-4f41-acb4-4cd03c913cdf" containerName="manager" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056777 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166bed11-aeb3-4e6c-8adc-77904aee4228" containerName="redis" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056783 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="166bed11-aeb3-4e6c-8adc-77904aee4228" containerName="redis" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056791 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="feast-init" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056796 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="feast-init" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056804 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="registry" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056809 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="registry" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056817 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="registry" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056822 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="registry" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056834 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="feast-init" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056839 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="feast-init" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056844 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="feast-apply" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056849 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="feast-apply" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056886 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee087239-2846-4685-86aa-cd60e9f62cc9" containerName="postgres" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056892 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="online" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056898 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="registry" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056906 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="166bed11-aeb3-4e6c-8adc-77904aee4228" containerName="redis" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056911 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cceb9ff8-0669-421d-904b-fd48e4abd21e" containerName="registry" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056916 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8140b4f0-6fff-4f41-acb4-4cd03c913cdf" containerName="manager" Sep 09 15:06:48.056944 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.056921 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a46bae8-7e65-4dd2-b66d-e2fce80e5487" containerName="online" Sep 09 15:06:48.059631 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.059610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:06:48.061850 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.061820 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rdv72\"/\"kube-root-ca.crt\"" Sep 09 15:06:48.061966 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.061847 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rdv72\"/\"openshift-service-ca.crt\"" Sep 09 15:06:48.062483 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.062463 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rdv72\"/\"default-dockercfg-66qv8\"" Sep 09 15:06:48.068003 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.067980 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdv72/must-gather-2ch79"] Sep 09 15:06:48.176309 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.176276 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgsct\" (UniqueName: \"kubernetes.io/projected/07e575cb-8e5d-4a04-9033-7c2d436a43cc-kube-api-access-mgsct\") pod \"must-gather-2ch79\" (UID: \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\") " pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:06:48.176494 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.176316 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07e575cb-8e5d-4a04-9033-7c2d436a43cc-must-gather-output\") pod \"must-gather-2ch79\" (UID: \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\") " pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:06:48.276797 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.276760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsct\" (UniqueName: \"kubernetes.io/projected/07e575cb-8e5d-4a04-9033-7c2d436a43cc-kube-api-access-mgsct\") pod \"must-gather-2ch79\" (UID: \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\") " pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:06:48.276961 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.276807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07e575cb-8e5d-4a04-9033-7c2d436a43cc-must-gather-output\") pod \"must-gather-2ch79\" (UID: \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\") " pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:06:48.277100 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.277086 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07e575cb-8e5d-4a04-9033-7c2d436a43cc-must-gather-output\") pod \"must-gather-2ch79\" (UID: \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\") " pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:06:48.285767 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.285734 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsct\" (UniqueName: \"kubernetes.io/projected/07e575cb-8e5d-4a04-9033-7c2d436a43cc-kube-api-access-mgsct\") pod \"must-gather-2ch79\" (UID: \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\") " pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:06:48.368543 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.368440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:06:48.635814 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.635741 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdv72/must-gather-2ch79"] Sep 09 15:06:48.638907 ip-10-0-130-173 kubenswrapper[2578]: W0909 15:06:48.638879 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07e575cb_8e5d_4a04_9033_7c2d436a43cc.slice/crio-62e3f0949730b81496fbe5dfe355f300eaf36063a60f90d9c408910fd8a86fcd WatchSource:0}: Error finding container 62e3f0949730b81496fbe5dfe355f300eaf36063a60f90d9c408910fd8a86fcd: Status 404 returned error can't find the container with id 62e3f0949730b81496fbe5dfe355f300eaf36063a60f90d9c408910fd8a86fcd Sep 09 15:06:48.641128 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:48.641111 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 15:06:49.315284 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:49.315244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdv72/must-gather-2ch79" event={"ID":"07e575cb-8e5d-4a04-9033-7c2d436a43cc","Type":"ContainerStarted","Data":"62e3f0949730b81496fbe5dfe355f300eaf36063a60f90d9c408910fd8a86fcd"} Sep 09 15:06:55.350438 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:55.350396 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdv72/must-gather-2ch79" event={"ID":"07e575cb-8e5d-4a04-9033-7c2d436a43cc","Type":"ContainerStarted","Data":"15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7"} Sep 09 15:06:55.350438 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:06:55.350445 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdv72/must-gather-2ch79" event={"ID":"07e575cb-8e5d-4a04-9033-7c2d436a43cc","Type":"ContainerStarted","Data":"13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788"} Sep 09 15:07:04.384836 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:04.384805 2578 generic.go:356] "Generic (PLEG): container finished" podID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" containerID="13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788" exitCode=0 Sep 09 15:07:04.385175 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:04.384863 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdv72/must-gather-2ch79" event={"ID":"07e575cb-8e5d-4a04-9033-7c2d436a43cc","Type":"ContainerDied","Data":"13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788"} Sep 09 15:07:04.385175 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:04.385159 2578 scope.go:117] "RemoveContainer" containerID="13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788" Sep 09 15:07:05.361261 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:05.361226 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rdv72_must-gather-2ch79_07e575cb-8e5d-4a04-9033-7c2d436a43cc/gather/0.log" Sep 09 15:07:08.904125 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:08.904092 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k2jgj_ddac112e-73f0-4043-a739-7dd8b01527aa/global-pull-secret-syncer/0.log" Sep 09 15:07:09.104867 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:09.104815 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rr629_b67132ad-d647-4474-84cb-5991ab2ab883/konnectivity-agent/0.log" Sep 09 15:07:09.127593 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:09.127568 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-173.ec2.internal_ce69d8fb05d3b4f877b142f86f624c3c/haproxy/0.log" Sep 09 15:07:10.668800 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:10.668765 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-rdv72/must-gather-2ch79"] Sep 09 15:07:10.669316 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:10.668974 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-rdv72/must-gather-2ch79" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" containerName="copy" containerID="cri-o://15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7" gracePeriod=2 Sep 09 15:07:10.670764 ip-10-0-130-173 kubenswrapper[2578]: E0909 15:07:10.670724 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2ch79\" is forbidden: User \"system:node:ip-10-0-130-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rdv72\": no relationship found between node 'ip-10-0-130-173.ec2.internal' and this object" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:07:10.670935 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:10.670917 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-rdv72/must-gather-2ch79"] Sep 09 15:07:10.891905 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:10.891883 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rdv72_must-gather-2ch79_07e575cb-8e5d-4a04-9033-7c2d436a43cc/copy/0.log" Sep 09 15:07:10.893420 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:10.893403 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:07:10.895378 ip-10-0-130-173 kubenswrapper[2578]: E0909 15:07:10.895355 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2ch79\" is forbidden: User \"system:node:ip-10-0-130-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rdv72\": no relationship found between node 'ip-10-0-130-173.ec2.internal' and this object" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:07:10.980086 ip-10-0-130-173 kubenswrapper[2578]: E0909 15:07:10.980055 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2ch79\" is forbidden: User \"system:node:ip-10-0-130-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rdv72\": no relationship found between node 'ip-10-0-130-173.ec2.internal' and this object" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:07:11.057570 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.057540 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgsct\" (UniqueName: \"kubernetes.io/projected/07e575cb-8e5d-4a04-9033-7c2d436a43cc-kube-api-access-mgsct\") pod \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\" (UID: \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\") " Sep 09 15:07:11.057570 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.057575 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07e575cb-8e5d-4a04-9033-7c2d436a43cc-must-gather-output\") pod \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\" (UID: \"07e575cb-8e5d-4a04-9033-7c2d436a43cc\") " Sep 09 15:07:11.057916 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.057896 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e575cb-8e5d-4a04-9033-7c2d436a43cc-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "07e575cb-8e5d-4a04-9033-7c2d436a43cc" (UID: "07e575cb-8e5d-4a04-9033-7c2d436a43cc"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 15:07:11.059830 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.059800 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e575cb-8e5d-4a04-9033-7c2d436a43cc-kube-api-access-mgsct" (OuterVolumeSpecName: "kube-api-access-mgsct") pod "07e575cb-8e5d-4a04-9033-7c2d436a43cc" (UID: "07e575cb-8e5d-4a04-9033-7c2d436a43cc"). InnerVolumeSpecName "kube-api-access-mgsct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 15:07:11.158309 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.158276 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgsct\" (UniqueName: \"kubernetes.io/projected/07e575cb-8e5d-4a04-9033-7c2d436a43cc-kube-api-access-mgsct\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 15:07:11.158309 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.158303 2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/07e575cb-8e5d-4a04-9033-7c2d436a43cc-must-gather-output\") on node \"ip-10-0-130-173.ec2.internal\" DevicePath \"\"" Sep 09 15:07:11.412272 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.412197 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rdv72_must-gather-2ch79_07e575cb-8e5d-4a04-9033-7c2d436a43cc/copy/0.log" Sep 09 15:07:11.412595 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.412569 2578 generic.go:356] "Generic (PLEG): container finished" podID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" containerID="15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7" exitCode=143 Sep 09 15:07:11.412666 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.412617 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdv72/must-gather-2ch79" Sep 09 15:07:11.412712 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.412677 2578 scope.go:117] "RemoveContainer" containerID="15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7" Sep 09 15:07:11.420705 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.420688 2578 scope.go:117] "RemoveContainer" containerID="13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788" Sep 09 15:07:11.432100 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.432084 2578 scope.go:117] "RemoveContainer" containerID="15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7" Sep 09 15:07:11.432367 ip-10-0-130-173 kubenswrapper[2578]: E0909 15:07:11.432334 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7\": container with ID starting with 15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7 not found: ID does not exist" containerID="15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7" Sep 09 15:07:11.432425 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.432376 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7"} err="failed to get container status \"15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7\": rpc error: code = NotFound desc = could not find container \"15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7\": container with ID starting with 15f0a74218d16ba897caf2ba76c729c7475fdf297bfde22be15321bb06391cc7 not found: ID does not exist" Sep 09 15:07:11.432425 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.432393 2578 scope.go:117] "RemoveContainer" containerID="13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788" Sep 09 15:07:11.432650 ip-10-0-130-173 kubenswrapper[2578]: E0909 15:07:11.432630 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788\": container with ID starting with 13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788 not found: ID does not exist" containerID="13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788" Sep 09 15:07:11.432701 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:11.432659 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788"} err="failed to get container status \"13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788\": rpc error: code = NotFound desc = could not find container \"13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788\": container with ID starting with 13eaae2fabb03135f86a2206de9728c376f6941f27e45cbafa9523da5af93788 not found: ID does not exist" Sep 09 15:07:12.981535 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:12.981507 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" path="/var/lib/kubelet/pods/07e575cb-8e5d-4a04-9033-7c2d436a43cc/volumes" Sep 09 15:07:13.303092 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:13.303058 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-9tkzv_52d3841d-c086-4a5a-99dd-6f98adffbfd9/cluster-monitoring-operator/0.log" Sep 09 15:07:13.472960 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:13.472924 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2dlbn_14079d71-4d00-41e3-b4f0-efab1cf10451/node-exporter/0.log" Sep 09 15:07:13.494995 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:13.494967 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2dlbn_14079d71-4d00-41e3-b4f0-efab1cf10451/kube-rbac-proxy/0.log" Sep 09 15:07:13.516905 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:13.516874 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2dlbn_14079d71-4d00-41e3-b4f0-efab1cf10451/init-textfile/0.log" Sep 09 15:07:13.961273 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:13.961232 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-xzq54_6e992a04-0505-4355-aa11-b1038a72882f/prometheus-operator/0.log" Sep 09 15:07:13.977146 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:13.977116 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-xzq54_6e992a04-0505-4355-aa11-b1038a72882f/kube-rbac-proxy/0.log" Sep 09 15:07:13.998652 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:13.998622 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-kb94l_55b8f528-bf00-4509-bb5a-7ef7083dfb0a/prometheus-operator-admission-webhook/0.log" Sep 09 15:07:15.952330 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:15.952298 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/1.log" Sep 09 15:07:15.960112 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:15.960056 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-n7g7g_96887fc2-e8bc-4ae0-b45d-96f012fbcf07/console-operator/2.log" Sep 09 15:07:16.807997 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:16.807969 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-gj289_f739c355-396f-4032-86b7-3ad0ccc92c48/volume-data-source-validator/0.log" Sep 09 15:07:17.553914 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:17.553884 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9v8sj_e2994b71-2a39-42b9-bd8c-1b55c03259b0/dns/0.log" Sep 09 15:07:17.571493 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:17.571468 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9v8sj_e2994b71-2a39-42b9-bd8c-1b55c03259b0/kube-rbac-proxy/0.log" Sep 09 15:07:17.675560 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:17.675533 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qhwvw_fffaf1ff-bc62-4ce6-a5ca-a45d536a4217/dns-node-resolver/0.log" Sep 09 15:07:18.062020 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.061991 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4v9k9_8b9fd951-66f4-459d-966d-6565d97cec02/node-ca/0.log" Sep 09 15:07:18.676175 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.676141 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6"] Sep 09 15:07:18.676556 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.676402 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" containerName="copy" Sep 09 15:07:18.676556 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.676414 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" containerName="copy" Sep 09 15:07:18.676556 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.676431 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" containerName="gather" Sep 09 15:07:18.676556 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.676438 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" containerName="gather" Sep 09 15:07:18.676556 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.676481 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" containerName="gather" Sep 09 15:07:18.676556 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.676487 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e575cb-8e5d-4a04-9033-7c2d436a43cc" containerName="copy" Sep 09 15:07:18.679671 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.679648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.681928 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.681908 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjq67\"/\"kube-root-ca.crt\"" Sep 09 15:07:18.682065 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.681972 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjq67\"/\"openshift-service-ca.crt\"" Sep 09 15:07:18.682751 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.682731 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sjq67\"/\"default-dockercfg-fnk76\"" Sep 09 15:07:18.687293 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.687273 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6"] Sep 09 15:07:18.762073 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.762039 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-594c74b48d-2brjb_712c3b38-ef28-4237-8de9-1d9a6058866c/router/0.log" Sep 09 15:07:18.808108 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.808079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-proc\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.808280 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.808112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-sys\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.808280 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.808138 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr6zl\" (UniqueName: \"kubernetes.io/projected/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-kube-api-access-tr6zl\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.808280 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.808219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-lib-modules\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.808280 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.808270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-podres\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.909091 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.909059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-proc\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.909226 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.909096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-sys\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.909226 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.909120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6zl\" (UniqueName: \"kubernetes.io/projected/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-kube-api-access-tr6zl\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.909226 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.909177 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-proc\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.909226 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.909187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-sys\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.909226 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.909222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-lib-modules\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.909448 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.909255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-podres\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.909448 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.909390 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-podres\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.909448 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.909393 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-lib-modules\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.916016 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.915995 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6zl\" (UniqueName: \"kubernetes.io/projected/9d769bf6-5a9f-4687-98c8-45d23fe3e06a-kube-api-access-tr6zl\") pod \"perf-node-gather-daemonset-2vkn6\" (UID: \"9d769bf6-5a9f-4687-98c8-45d23fe3e06a\") " pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:18.989024 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:18.989003 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:19.102967 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:19.102942 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5ntb8_9ef9739c-da6d-450b-be72-a2d7c60b9027/serve-healthcheck-canary/0.log" Sep 09 15:07:19.256254 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:19.256181 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6"] Sep 09 15:07:19.259068 ip-10-0-130-173 kubenswrapper[2578]: W0909 15:07:19.259044 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9d769bf6_5a9f_4687_98c8_45d23fe3e06a.slice/crio-a04ccf1fee02af83debcdc18cc82e24524e8cfffee3f85b852e42e7a7e926557 WatchSource:0}: Error finding container a04ccf1fee02af83debcdc18cc82e24524e8cfffee3f85b852e42e7a7e926557: Status 404 returned error can't find the container with id a04ccf1fee02af83debcdc18cc82e24524e8cfffee3f85b852e42e7a7e926557 Sep 09 15:07:19.432759 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:19.432729 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6hltd_a70e9268-21ad-48cc-b7ef-30a315996603/kube-rbac-proxy/0.log" Sep 09 15:07:19.441402 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:19.441374 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" event={"ID":"9d769bf6-5a9f-4687-98c8-45d23fe3e06a","Type":"ContainerStarted","Data":"2ca36a5a36083e779b26417f50e22a3925878df5a8b1a1f4bd4359ec754c6f0a"} Sep 09 15:07:19.441549 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:19.441422 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" event={"ID":"9d769bf6-5a9f-4687-98c8-45d23fe3e06a","Type":"ContainerStarted","Data":"a04ccf1fee02af83debcdc18cc82e24524e8cfffee3f85b852e42e7a7e926557"} Sep 09 15:07:19.441549 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:19.441446 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:19.457185 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:19.457147 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6hltd_a70e9268-21ad-48cc-b7ef-30a315996603/exporter/0.log" Sep 09 15:07:19.457321 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:19.457287 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" podStartSLOduration=1.45727306 podStartE2EDuration="1.45727306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 15:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 15:07:19.456712737 +0000 UTC m=+2039.123518784" watchObservedRunningTime="2026-09-09 15:07:19.45727306 +0000 UTC m=+2039.124079125" Sep 09 15:07:19.477042 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:19.477018 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6hltd_a70e9268-21ad-48cc-b7ef-30a315996603/extractor/0.log" Sep 09 15:07:24.945242 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:24.945210 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rl6h9_af356e02-73e3-4111-a82b-e1462c92982b/kube-multus-additional-cni-plugins/0.log" Sep 09 15:07:24.963811 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:24.963786 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rl6h9_af356e02-73e3-4111-a82b-e1462c92982b/egress-router-binary-copy/0.log" Sep 09 15:07:24.981919 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:24.981897 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rl6h9_af356e02-73e3-4111-a82b-e1462c92982b/cni-plugins/0.log" Sep 09 15:07:24.999782 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:24.999756 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rl6h9_af356e02-73e3-4111-a82b-e1462c92982b/bond-cni-plugin/0.log" Sep 09 15:07:25.017855 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.017834 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rl6h9_af356e02-73e3-4111-a82b-e1462c92982b/routeoverride-cni/0.log" Sep 09 15:07:25.035996 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.035973 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rl6h9_af356e02-73e3-4111-a82b-e1462c92982b/whereabouts-cni-bincopy/0.log" Sep 09 15:07:25.054707 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.054690 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rl6h9_af356e02-73e3-4111-a82b-e1462c92982b/whereabouts-cni/0.log" Sep 09 15:07:25.090488 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.090467 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vsqcx_e8fcd125-d61b-4c71-a1a2-b7bb507101b1/kube-multus/0.log" Sep 09 15:07:25.144844 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.144825 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fs9bf_c566cce4-7251-42bf-8696-f2505562d073/network-metrics-daemon/0.log" Sep 09 15:07:25.161076 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.161053 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fs9bf_c566cce4-7251-42bf-8696-f2505562d073/kube-rbac-proxy/0.log" Sep 09 15:07:25.454618 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.454596 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sjq67/perf-node-gather-daemonset-2vkn6" Sep 09 15:07:25.886451 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.886422 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66bb4_bfe14675-8e24-4434-a00e-e9fb8f83e0a6/ovn-controller/0.log" Sep 09 15:07:25.900554 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.900523 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66bb4_bfe14675-8e24-4434-a00e-e9fb8f83e0a6/ovn-acl-logging/0.log" Sep 09 15:07:25.928641 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.928599 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66bb4_bfe14675-8e24-4434-a00e-e9fb8f83e0a6/kube-rbac-proxy-node/0.log" Sep 09 15:07:25.952893 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.952866 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66bb4_bfe14675-8e24-4434-a00e-e9fb8f83e0a6/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 15:07:25.969476 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.969454 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66bb4_bfe14675-8e24-4434-a00e-e9fb8f83e0a6/northd/0.log" Sep 09 15:07:25.987613 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:25.987588 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66bb4_bfe14675-8e24-4434-a00e-e9fb8f83e0a6/nbdb/0.log" Sep 09 15:07:26.005342 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:26.005319 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66bb4_bfe14675-8e24-4434-a00e-e9fb8f83e0a6/sbdb/0.log" Sep 09 15:07:26.053465 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:26.053440 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66bb4_bfe14675-8e24-4434-a00e-e9fb8f83e0a6/ovnkube-controller/0.log" Sep 09 15:07:27.283271 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:27.283234 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5sm4r_e2f50ed1-8a0b-4bff-a86c-c46232240402/network-check-target-container/0.log" Sep 09 15:07:28.191331 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:28.191307 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-z5gj9_b37cc860-a60b-44a7-b363-913eb3128dc7/iptables-alerter/0.log" Sep 09 15:07:28.837215 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:28.837182 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-nf9q6_54a4de0c-d5f5-40d8-bdfa-b90b1003a96c/tuned/0.log" Sep 09 15:07:30.714672 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:30.714580 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-qfvdv_e0ce45bb-2ee5-4e19-92ef-134ca514e125/cluster-samples-operator/0.log" Sep 09 15:07:30.731921 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:30.731897 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-qfvdv_e0ce45bb-2ee5-4e19-92ef-134ca514e125/cluster-samples-operator-watch/0.log" Sep 09 15:07:31.804248 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:31.804215 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-2mw8h_ba32ff86-2186-4f15-84f4-8b2c63867723/service-ca-operator/1.log" Sep 09 15:07:31.805844 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:31.805823 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-2mw8h_ba32ff86-2186-4f15-84f4-8b2c63867723/service-ca-operator/0.log" Sep 09 15:07:32.670053 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:32.670027 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-p87x9_f2fa5ef4-acc5-488f-9da4-1d8b94103656/csi-driver/0.log" Sep 09 15:07:32.703047 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:32.703015 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-p87x9_f2fa5ef4-acc5-488f-9da4-1d8b94103656/csi-node-driver-registrar/0.log" Sep 09 15:07:32.740110 ip-10-0-130-173 kubenswrapper[2578]: I0909 15:07:32.740074 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-p87x9_f2fa5ef4-acc5-488f-9da4-1d8b94103656/csi-liveness-probe/0.log"