Sep 10 09:14:20.705557 ip-10-0-139-170 systemd[1]: Starting Kubernetes Kubelet... Sep 10 09:14:21.085163 ip-10-0-139-170 kubenswrapper[2546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 09:14:21.085163 ip-10-0-139-170 kubenswrapper[2546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 09:14:21.085163 ip-10-0-139-170 kubenswrapper[2546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 09:14:21.085163 ip-10-0-139-170 kubenswrapper[2546]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 09:14:21.085163 ip-10-0-139-170 kubenswrapper[2546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 09:14:21.085778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.085226 2546 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 09:14:21.089751 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089737 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 09:14:21.089751 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089751 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089755 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089758 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089761 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089764 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089767 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089770 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089773 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089776 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089780 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089783 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089785 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089788 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089791 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089793 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089796 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089799 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089807 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089810 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089813 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 09:14:21.089816 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089815 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089818 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089822 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089826 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089830 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089833 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089835 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089838 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089840 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089843 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089846 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089848 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089851 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089853 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089856 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089859 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089861 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089864 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089866 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 09:14:21.090426 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089868 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089871 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089873 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089875 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089878 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089880 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089883 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089885 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089888 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089890 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089893 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089896 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089898 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089901 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089904 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089907 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089910 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089912 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089915 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089918 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 09:14:21.090894 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089920 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089923 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089925 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089927 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089930 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089932 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089935 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089938 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089940 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089943 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089945 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089948 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089950 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089953 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089955 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089958 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089960 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089963 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089965 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089968 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 09:14:21.091392 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089970 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089975 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089978 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089982 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089986 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089989 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089992 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089995 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.089998 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090000 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090003 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090005 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090009 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090013 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090016 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090018 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090021 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090023 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.090027 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 09:14:21.091873 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091523 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091532 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091535 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091541 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091545 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091548 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091551 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091554 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091557 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091560 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091562 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091565 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091568 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091571 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091574 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091577 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091580 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091583 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091587 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 09:14:21.092342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091589 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091592 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091595 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091597 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091600 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091602 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091605 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091607 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091610 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091612 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091615 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091619 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091623 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091626 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091628 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091631 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091633 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091636 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091639 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091641 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 09:14:21.092851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091644 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091646 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091649 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091652 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091654 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091657 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091659 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091662 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091665 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091667 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091670 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091677 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091680 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091682 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091685 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091687 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091690 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091693 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091695 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091698 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 09:14:21.093345 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091700 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091703 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091705 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091708 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091710 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091713 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091715 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091718 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091720 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091723 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091725 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091728 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091730 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091732 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091735 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091737 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091740 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091743 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091745 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091748 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 09:14:21.093830 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091752 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091755 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091757 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091760 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091768 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091771 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091773 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091776 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091779 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091781 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091785 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091788 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091791 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091795 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091797 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091800 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091803 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091806 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091808 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.091811 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 09:14:21.094343 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092172 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092177 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092180 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092182 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092185 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092188 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092190 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092193 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092196 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092198 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092202 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092204 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092207 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092210 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092212 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092215 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092218 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092221 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092224 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092226 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 09:14:21.094832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092229 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092231 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092234 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092236 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092239 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092241 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092244 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092246 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092249 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092251 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092254 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092256 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092259 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092262 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092264 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092267 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092270 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092273 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092275 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092291 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 09:14:21.095352 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092294 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092297 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092299 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092302 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092305 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092307 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092310 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092313 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092316 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092319 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092322 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092324 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092327 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092331 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092334 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092337 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092342 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092345 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092347 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 09:14:21.095837 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092350 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092353 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092355 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092358 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092360 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092363 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092366 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092368 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092371 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092373 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092376 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092378 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092381 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092383 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092386 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092388 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092392 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092395 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092398 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092401 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 09:14:21.096308 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092403 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092406 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092409 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092419 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092422 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092424 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092427 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092430 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092432 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092435 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092438 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092440 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092443 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092445 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092449 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092452 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092455 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092458 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092461 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 09:14:21.096787 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.092463 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092545 2546 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092577 2546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092586 2546 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092590 2546 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092594 2546 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092598 2546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092601 2546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092607 2546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092609 2546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092612 2546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092616 2546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092619 2546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092622 2546 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092625 2546 flags.go:64] FLAG: --cgroup-root="" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092627 2546 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092630 2546 flags.go:64] FLAG: --client-ca-file="" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092634 2546 flags.go:64] FLAG: --cloud-provider="external" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092637 2546 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092644 2546 flags.go:64] FLAG: --cluster-domain="" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092646 2546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092649 2546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092653 2546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092656 2546 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 09:14:21.097245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092659 2546 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092662 2546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092665 2546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092669 2546 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092671 2546 flags.go:64] FLAG: --contention-profiling="false" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092674 2546 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092677 2546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092680 2546 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092683 2546 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092687 2546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092690 2546 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092692 2546 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092695 2546 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092698 2546 flags.go:64] FLAG: --enable-server="true" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092701 2546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092705 2546 flags.go:64] FLAG: --event-burst="100" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092711 2546 flags.go:64] FLAG: --event-qps="50" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092714 2546 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092716 2546 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092719 2546 flags.go:64] FLAG: --eviction-hard="" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092723 2546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092726 2546 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092728 2546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092731 2546 flags.go:64] FLAG: --eviction-soft="" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092734 2546 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 09:14:21.097823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092737 2546 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092740 2546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092743 2546 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092746 2546 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092749 2546 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092751 2546 flags.go:64] FLAG: --feature-gates="" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092755 2546 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092758 2546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092760 2546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092763 2546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092766 2546 flags.go:64] FLAG: --healthz-port="10248" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092769 2546 flags.go:64] FLAG: --help="false" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092772 2546 flags.go:64] FLAG: --hostname-override="ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092775 2546 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092778 2546 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092780 2546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092783 2546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092787 2546 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092789 2546 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092793 2546 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092795 2546 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092798 2546 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092801 2546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092805 2546 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 09:14:21.098529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092808 2546 flags.go:64] FLAG: --kube-reserved="" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092811 2546 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092814 2546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092817 2546 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092819 2546 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092822 2546 flags.go:64] FLAG: --lock-file="" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092825 2546 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092828 2546 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092830 2546 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092835 2546 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092839 2546 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092841 2546 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092844 2546 flags.go:64] FLAG: --logging-format="text" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092847 2546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092851 2546 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092853 2546 flags.go:64] FLAG: --manifest-url="" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092856 2546 flags.go:64] FLAG: --manifest-url-header="" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092860 2546 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092864 2546 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092867 2546 flags.go:64] FLAG: --max-pods="110" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092870 2546 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092873 2546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092876 2546 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092879 2546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092882 2546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 09:14:21.099145 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092884 2546 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092887 2546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092894 2546 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092897 2546 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092899 2546 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092902 2546 flags.go:64] FLAG: --pod-cidr="" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092906 2546 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092911 2546 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092914 2546 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092917 2546 flags.go:64] FLAG: --pods-per-core="0" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092920 2546 flags.go:64] FLAG: --port="10250" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092923 2546 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092926 2546 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-006338cf2d9e01c89" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092929 2546 flags.go:64] FLAG: --qos-reserved="" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092932 2546 flags.go:64] FLAG: --read-only-port="10255" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092934 2546 flags.go:64] FLAG: --register-node="true" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092937 2546 flags.go:64] FLAG: --register-with-taints="" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092942 2546 flags.go:64] FLAG: --registry-burst="10" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092945 2546 flags.go:64] FLAG: --registry-qps="5" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092948 2546 flags.go:64] FLAG: --reserved-cpus="" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092951 2546 flags.go:64] FLAG: --reserved-memory="" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092954 2546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092957 2546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092960 2546 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092963 2546 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 09:14:21.099780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092966 2546 flags.go:64] FLAG: --runonce="false" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092969 2546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092972 2546 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092975 2546 flags.go:64] FLAG: --seccomp-default="false" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092977 2546 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092980 2546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092983 2546 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092986 2546 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092989 2546 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092992 2546 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092995 2546 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.092998 2546 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093000 2546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093004 2546 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093008 2546 flags.go:64] FLAG: --system-cgroups="" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093011 2546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093016 2546 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093019 2546 flags.go:64] FLAG: --tls-cert-file="" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093022 2546 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093025 2546 flags.go:64] FLAG: --tls-min-version="" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093028 2546 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093030 2546 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093033 2546 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093036 2546 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093040 2546 flags.go:64] FLAG: --v="2" Sep 10 09:14:21.100483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093044 2546 flags.go:64] FLAG: --version="false" Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093048 2546 flags.go:64] FLAG: --vmodule="" Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093051 2546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.093054 2546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093160 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093164 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093167 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093170 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093173 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093176 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093179 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093181 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093184 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093187 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093190 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093192 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093195 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093198 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093200 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093203 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093206 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 09:14:21.101387 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093209 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093212 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093215 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093217 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093220 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093222 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093225 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093228 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093230 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093233 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093237 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093239 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093242 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093245 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093248 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093250 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093253 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093256 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093261 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093265 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 09:14:21.101931 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093269 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093271 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093274 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093289 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093293 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093297 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093300 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093303 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093306 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093309 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093312 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093314 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093319 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093322 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093324 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093327 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093330 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093332 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093335 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 09:14:21.102491 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093337 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093340 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093343 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093346 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093349 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093352 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093355 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093358 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093361 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093365 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093368 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093371 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093373 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093376 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093379 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093382 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093384 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093387 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093390 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093392 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 09:14:21.102969 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093394 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093397 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093400 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093402 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093405 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093409 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093411 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093414 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093416 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093419 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093422 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093425 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093427 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093430 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093432 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093437 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093439 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093442 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093444 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093447 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 09:14:21.103542 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093451 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093454 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.093458 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.094033 2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.102254 2546 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.102268 2546 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.103906 2546 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.103918 2546 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.103978 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.103983 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.103986 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.103989 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.103992 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.103995 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.103998 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 09:14:21.104029 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104000 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104003 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104006 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104009 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104011 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104014 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104016 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104019 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104022 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104025 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104028 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104031 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104034 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104036 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104039 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104042 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104046 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104050 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104053 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 09:14:21.104410 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104056 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104059 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104062 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104065 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104068 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104071 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104074 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104077 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104079 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104082 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104085 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104087 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104089 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104092 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104095 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104098 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104100 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104103 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104106 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104108 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 09:14:21.104875 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104111 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104114 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104117 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104119 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104122 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104124 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104127 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104129 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104132 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104134 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104138 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104141 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104144 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104147 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104149 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104151 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104154 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104157 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104159 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104162 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 09:14:21.105371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104164 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104167 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104170 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104173 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104175 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104177 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104180 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104182 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104185 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104188 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104190 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104192 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104195 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104205 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104208 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104211 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104213 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104216 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104218 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104221 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 09:14:21.105876 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104223 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104226 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104229 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104231 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104234 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104237 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104239 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104242 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104244 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104246 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104251 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104254 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104257 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.104262 2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 09:14:21.106371 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104386 2546 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104392 2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104395 2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104399 2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104403 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104406 2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104409 2546 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104412 2546 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104415 2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104418 2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104420 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104423 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104432 2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104434 2546 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104437 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104440 2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104442 2546 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104445 2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104448 2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104450 2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 09:14:21.106719 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104453 2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104456 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104458 2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104461 2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104463 2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104466 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104469 2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104473 2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104476 2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104478 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104481 2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104484 2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104487 2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104490 2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104493 2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104495 2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104498 2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104500 2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104503 2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 09:14:21.107205 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104506 2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104510 2546 feature_gate.go:328] unrecognized feature gate: Example Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104512 2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104515 2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104517 2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104520 2546 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104530 2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104532 2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104535 2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104537 2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104540 2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104542 2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104545 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104547 2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104550 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104552 2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104555 2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104557 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104560 2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104562 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 09:14:21.107684 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104565 2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104567 2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104570 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104572 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104575 2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104578 2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104580 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104583 2546 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104585 2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104588 2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104590 2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104593 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104595 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104598 2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104600 2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104603 2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104605 2546 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104608 2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104610 2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 09:14:21.108164 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104618 2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104621 2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104623 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104626 2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104628 2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104631 2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104633 2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104636 2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104639 2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104641 2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104644 2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104646 2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104648 2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104651 2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104654 2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104656 2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104658 2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104663 2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104665 2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104668 2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 09:14:21.108663 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:21.104671 2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 09:14:21.109138 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.104676 2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 09:14:21.109138 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.104766 2546 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 09:14:21.109138 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.106681 2546 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 09:14:21.109138 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.107583 2546 server.go:1019] "Starting client certificate rotation" Sep 10 09:14:21.109138 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.107681 2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 09:14:21.109138 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.107710 2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 09:14:21.110446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.110433 2546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 09:14:21.111702 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.111681 2546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 09:14:21.122723 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.122707 2546 log.go:25] "Validated CRI v1 runtime API" Sep 10 09:14:21.125505 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.125491 2546 log.go:25] "Validated CRI v1 image API" Sep 10 09:14:21.126310 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.126295 2546 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 09:14:21.128741 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.128724 2546 fs.go:135] Filesystem UUIDs: map[4b93d48d-57c3-45ba-a827-bcad99734833:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b6476cb3-b91a-4da1-a778-c56e4d9b9d5f:/dev/nvme0n1p4] Sep 10 09:14:21.128787 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.128740 2546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 09:14:21.134412 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.134300 2546 manager.go:216] Machine: {Timestamp:2026-09-10 09:14:21.133067709 +0000 UTC m=+0.314756766 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100152 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2322b0a477880a3eabc4a2dc0828e4 SystemUUID:ec2322b0-a477-880a-3eab-c4a2dc0828e4 BootID:29c9df0e-d36c-4a7e-82c9-1a27da09e634 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:63:5d:91:0c:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:63:5d:91:0c:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:c5:ac:a1:f9:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 09:14:21.134412 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.134402 2546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 09:14:21.134525 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.134471 2546 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 09:14:21.135178 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.135160 2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 09:14:21.137855 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.137827 2546 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 09:14:21.138019 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.137867 2546 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-170.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 09:14:21.139031 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.139019 2546 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 09:14:21.139099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.139050 2546 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 09:14:21.139099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.139061 2546 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 09:14:21.139823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.139810 2546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 09:14:21.140868 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.140856 2546 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 09:14:21.143342 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.143329 2546 state_mem.go:36] "Initialized new in-memory state store" Sep 10 09:14:21.143482 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.143470 2546 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 09:14:21.143544 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.143505 2546 kubelet.go:486] "Attempting to sync node with API server" Sep 10 09:14:21.143544 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.143518 2546 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 09:14:21.143544 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.143540 2546 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 09:14:21.143670 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.143561 2546 kubelet.go:398] "Adding apiserver pod source" Sep 10 09:14:21.143670 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.143572 2546 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 09:14:21.143949 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.143935 2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 09:14:21.144013 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.143960 2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 09:14:21.147706 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.147677 2546 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 09:14:21.149966 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.149870 2546 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 09:14:21.150017 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.149982 2546 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 09:14:21.150136 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.150126 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 09:14:21.150173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.150141 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 09:14:21.150173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.150148 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 09:14:21.150173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.150153 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 09:14:21.150173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.150161 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 09:14:21.150173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.150166 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 09:14:21.151357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.151346 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 09:14:21.151357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.151357 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 09:14:21.151473 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.151367 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 09:14:21.151473 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.151375 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 09:14:21.151473 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.151385 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 09:14:21.151473 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.151393 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 09:14:21.152134 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.152125 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 09:14:21.152134 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.152133 2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 09:14:21.155757 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.155743 2546 server.go:1268] "Started kubelet" Sep 10 09:14:21.155869 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.155807 2546 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 09:14:21.155923 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.155835 2546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 09:14:21.155970 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.155939 2546 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 09:14:21.156394 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.156376 2546 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-170.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 09:14:21.156394 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.156385 2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 09:14:21.156520 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.156484 2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 09:14:21.156557 ip-10-0-139-170 systemd[1]: Started Kubernetes Kubelet. Sep 10 09:14:21.156997 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.156976 2546 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 09:14:21.158232 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.158215 2546 server.go:310] "Adding debug handlers to kubelet server" Sep 10 09:14:21.161452 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.161434 2546 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 09:14:21.161532 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.161434 2546 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 09:14:21.161532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.161451 2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 09:14:21.161532 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.161525 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:21.161952 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.161933 2546 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 09:14:21.162046 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.161960 2546 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 09:14:21.162046 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.161929 2546 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 09:14:21.162148 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162078 2546 reconstruct.go:97] "Volume reconstruction finished" Sep 10 09:14:21.162148 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162086 2546 reconciler.go:29] "Reconciler: start to sync state" Sep 10 09:14:21.162246 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162174 2546 factory.go:55] Registering systemd factory Sep 10 09:14:21.162246 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162188 2546 factory.go:223] Registration of the systemd container factory successfully Sep 10 09:14:21.162435 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162373 2546 factory.go:153] Registering CRI-O factory Sep 10 09:14:21.162435 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162389 2546 factory.go:223] Registration of the crio container factory successfully Sep 10 09:14:21.162550 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162436 2546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 09:14:21.162550 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162453 2546 factory.go:103] Registering Raw factory Sep 10 09:14:21.162550 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162466 2546 manager.go:1196] Started watching for new ooms in manager Sep 10 09:14:21.163132 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.162813 2546 manager.go:349] Starting recovery of all containers Sep 10 09:14:21.163132 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.161796 2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-170.ec2.internal.18d3ead12f1d1290 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-170.ec2.internal,UID:ip-10-0-139-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-170.ec2.internal,},FirstTimestamp:2026-09-10 09:14:21.155717776 +0000 UTC m=+0.337406852,LastTimestamp:2026-09-10 09:14:21.155717776 +0000 UTC m=+0.337406852,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-170.ec2.internal,}" Sep 10 09:14:21.167148 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.167118 2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wj28w" Sep 10 09:14:21.170051 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.170026 2546 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 09:14:21.173628 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.173589 2546 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 09:14:21.174605 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.174579 2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wj28w" Sep 10 09:14:21.174753 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.174734 2546 manager.go:354] Recovery completed Sep 10 09:14:21.178865 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.178758 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 09:14:21.180849 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.180832 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Sep 10 09:14:21.180911 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.180864 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 09:14:21.180911 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.180878 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Sep 10 09:14:21.181412 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.181398 2546 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 09:14:21.181412 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.181409 2546 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 09:14:21.181412 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.181413 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 09:14:21.181561 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.181423 2546 state_mem.go:36] "Initialized new in-memory state store" Sep 10 09:14:21.182070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.182054 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Sep 10 09:14:21.182143 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.182085 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 09:14:21.182143 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.182106 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Sep 10 09:14:21.182548 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.182532 2546 policy_none.go:49] "None policy: Start" Sep 10 09:14:21.182607 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.182552 2546 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 09:14:21.182607 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.182566 2546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 09:14:21.182720 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.182645 2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-170.ec2.internal.18d3ead1309c8aae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-170.ec2.internal,UID:ip-10-0-139-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-170.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-170.ec2.internal,},FirstTimestamp:2026-09-10 09:14:21.180848814 +0000 UTC m=+0.362537872,LastTimestamp:2026-09-10 09:14:21.180848814 +0000 UTC m=+0.362537872,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-170.ec2.internal,}" Sep 10 09:14:21.183523 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.183510 2546 policy_none.go:47] "Start" Sep 10 09:14:21.210964 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.210948 2546 manager.go:337] "Starting Device Plugin manager" Sep 10 09:14:21.225487 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.210993 2546 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 09:14:21.225487 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.211012 2546 server.go:87] "Starting device plugin registration server" Sep 10 09:14:21.225487 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.211301 2546 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 09:14:21.225487 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.211313 2546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 09:14:21.225487 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.211427 2546 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 09:14:21.225487 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.211497 2546 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 09:14:21.225487 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.211509 2546 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 09:14:21.225487 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.211974 2546 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 09:14:21.225487 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.212019 2546 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:21.253836 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.253813 2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 09:14:21.254941 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.254919 2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 09:14:21.255019 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.254973 2546 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 09:14:21.255019 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.254997 2546 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 09:14:21.255133 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.255045 2546 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 09:14:21.257477 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.257459 2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 09:14:21.311677 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.311662 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 09:14:21.312389 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.312369 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Sep 10 09:14:21.312483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.312406 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 09:14:21.312483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.312421 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Sep 10 09:14:21.312483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.312445 2546 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.320333 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.320315 2546 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.320403 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.320334 2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-170.ec2.internal\": node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:21.339416 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.339363 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:21.355103 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.355080 2546 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal"] Sep 10 09:14:21.355170 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.355150 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 09:14:21.356360 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.356345 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Sep 10 09:14:21.356439 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.356368 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 09:14:21.356439 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.356381 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Sep 10 09:14:21.357449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.357436 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 09:14:21.357573 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.357553 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.357622 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.357579 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 09:14:21.358135 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.358118 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Sep 10 09:14:21.358193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.358144 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 09:14:21.358193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.358153 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Sep 10 09:14:21.358306 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.358122 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Sep 10 09:14:21.358306 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.358232 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 09:14:21.358306 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.358246 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Sep 10 09:14:21.359151 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.359133 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.359295 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.359157 2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 09:14:21.359795 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.359775 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientMemory" Sep 10 09:14:21.359876 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.359808 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 09:14:21.359876 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.359821 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeHasSufficientPID" Sep 10 09:14:21.362971 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.362950 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/050d88a1f009d75cf4482ef0a62462f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"050d88a1f009d75cf4482ef0a62462f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.363070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.362974 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/050d88a1f009d75cf4482ef0a62462f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"050d88a1f009d75cf4482ef0a62462f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.363070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.363001 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a131841e4a19fc7a8712e64d2e2b1c9d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-170.ec2.internal\" (UID: \"a131841e4a19fc7a8712e64d2e2b1c9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.372798 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.372775 2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-170.ec2.internal\" not found" node="ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.377005 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.376990 2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-170.ec2.internal\" not found" node="ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.439678 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.439662 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:21.463531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.463512 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/050d88a1f009d75cf4482ef0a62462f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"050d88a1f009d75cf4482ef0a62462f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.463634 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.463545 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/050d88a1f009d75cf4482ef0a62462f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"050d88a1f009d75cf4482ef0a62462f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.463634 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.463568 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a131841e4a19fc7a8712e64d2e2b1c9d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-170.ec2.internal\" (UID: \"a131841e4a19fc7a8712e64d2e2b1c9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.463634 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.463610 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/050d88a1f009d75cf4482ef0a62462f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"050d88a1f009d75cf4482ef0a62462f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.463634 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.463608 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/050d88a1f009d75cf4482ef0a62462f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal\" (UID: \"050d88a1f009d75cf4482ef0a62462f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.463791 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.463653 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a131841e4a19fc7a8712e64d2e2b1c9d-config\") pod \"kube-apiserver-proxy-ip-10-0-139-170.ec2.internal\" (UID: \"a131841e4a19fc7a8712e64d2e2b1c9d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.540394 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.540364 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:21.640938 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.640887 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:21.675468 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.675449 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.680030 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:21.680011 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Sep 10 09:14:21.740964 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.740939 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:21.841410 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.841390 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:21.941889 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:21.941836 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:22.042375 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:22.042350 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:22.108021 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.107994 2546 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 09:14:22.108489 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.108154 2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 09:14:22.142425 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:22.142399 2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-170.ec2.internal\" not found" Sep 10 09:14:22.176893 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.176862 2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 09:09:21 +0000 UTC" deadline="2028-03-12 13:59:41.288027655 +0000 UTC" Sep 10 09:14:22.176893 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.176894 2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13180h45m19.111138984s" Sep 10 09:14:22.212528 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.212473 2546 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 09:14:22.228926 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:22.228716 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050d88a1f009d75cf4482ef0a62462f4.slice/crio-33a51e92eb34ab6502bb464eaece56ecad823780d5110629c9b893557171c794 WatchSource:0}: Error finding container 33a51e92eb34ab6502bb464eaece56ecad823780d5110629c9b893557171c794: Status 404 returned error can't find the container with id 33a51e92eb34ab6502bb464eaece56ecad823780d5110629c9b893557171c794 Sep 10 09:14:22.229585 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:22.229560 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda131841e4a19fc7a8712e64d2e2b1c9d.slice/crio-e9dfa2e95d66090703ec9125f0169d07e46d0b1141344286295c8c1d58e7c6bb WatchSource:0}: Error finding container e9dfa2e95d66090703ec9125f0169d07e46d0b1141344286295c8c1d58e7c6bb: Status 404 returned error can't find the container with id e9dfa2e95d66090703ec9125f0169d07e46d0b1141344286295c8c1d58e7c6bb Sep 10 09:14:22.233827 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.233808 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 09:14:22.258015 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.257978 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" event={"ID":"050d88a1f009d75cf4482ef0a62462f4","Type":"ContainerStarted","Data":"33a51e92eb34ab6502bb464eaece56ecad823780d5110629c9b893557171c794"} Sep 10 09:14:22.259045 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.259025 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" event={"ID":"a131841e4a19fc7a8712e64d2e2b1c9d","Type":"ContainerStarted","Data":"e9dfa2e95d66090703ec9125f0169d07e46d0b1141344286295c8c1d58e7c6bb"} Sep 10 09:14:22.262446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.262421 2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" Sep 10 09:14:22.275413 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.275392 2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 09:14:22.278076 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.278064 2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" Sep 10 09:14:22.285634 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.285616 2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 09:14:22.373725 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.373699 2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 09:14:22.383172 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:22.383151 2546 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 09:14:23.145847 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.145574 2546 apiserver.go:52] "Watching apiserver" Sep 10 09:14:23.152006 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.151979 2546 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 09:14:23.152940 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.152521 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jr854","kube-system/konnectivity-agent-j9qc8","kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal","openshift-dns/node-resolver-mjg8s","openshift-image-registry/node-ca-htq4b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal","openshift-multus/multus-additional-cni-plugins-5tt2m","openshift-multus/multus-zwqd9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j","openshift-cluster-node-tuning-operator/tuned-dwbqv","openshift-multus/network-metrics-daemon-knj5j","openshift-network-diagnostics/network-check-target-b49hb","openshift-network-operator/iptables-alerter-ctr4v","openshift-ovn-kubernetes/ovnkube-node-xsql9"] Sep 10 09:14:23.155372 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.155243 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.157693 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.157645 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.160123 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.158576 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 09:14:23.160123 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.158826 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 09:14:23.160123 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.159072 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4wqzb\"" Sep 10 09:14:23.160123 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.159250 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 09:14:23.160123 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.159745 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-snbj4\"" Sep 10 09:14:23.160123 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.159966 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 09:14:23.160619 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.160237 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 09:14:23.163988 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.160973 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.163988 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.163266 2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 09:14:23.163988 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.163716 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.163988 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.163781 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:23.164874 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.164489 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 09:14:23.164874 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.164656 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 09:14:23.164874 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.164756 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jmv42\"" Sep 10 09:14:23.165195 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.165168 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.167247 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.167225 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.167988 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.167968 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 09:14:23.168101 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.168084 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 09:14:23.168244 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.168228 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 09:14:23.168325 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.168282 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ghbvg\"" Sep 10 09:14:23.169304 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.169271 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 09:14:23.170296 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.169670 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 09:14:23.170296 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.169938 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 09:14:23.170296 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.170097 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 09:14:23.170296 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.170143 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gr7pt\"" Sep 10 09:14:23.170296 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.170290 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 09:14:23.170589 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.170377 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.172098 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.172064 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:23.174339 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174301 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8615e2e1-f518-4237-960f-2eda0b516165-host\") pod \"node-ca-htq4b\" (UID: \"8615e2e1-f518-4237-960f-2eda0b516165\") " pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.174439 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174344 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-kubernetes\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.174439 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174393 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-sysctl-conf\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.174439 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174422 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.174600 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174447 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.174600 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174472 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-lib-modules\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.174600 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174510 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-var-lib-kubelet\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.174719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174679 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6k2r\" (UniqueName: \"kubernetes.io/projected/8615e2e1-f518-4237-960f-2eda0b516165-kube-api-access-k6k2r\") pod \"node-ca-htq4b\" (UID: \"8615e2e1-f518-4237-960f-2eda0b516165\") " pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.174719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174705 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-modprobe-d\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.174815 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174741 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6mw6\" (UniqueName: \"kubernetes.io/projected/0ed4f23e-1ad2-4346-8550-93565896543d-kube-api-access-k6mw6\") pod \"node-resolver-mjg8s\" (UID: \"0ed4f23e-1ad2-4346-8550-93565896543d\") " pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.174815 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174765 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce6a92df-68a8-49fa-acb8-3b40e6852595-dbus\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.174815 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174802 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-systemd\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.174922 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174844 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-sys\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.174922 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174851 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:23.174922 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174873 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-tuned\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.174922 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174911 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce6a92df-68a8-49fa-acb8-3b40e6852595-kubelet-config\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.174922 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.174910 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174953 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-sys-fs\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.174990 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-sysconfig\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175015 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ed4f23e-1ad2-4346-8550-93565896543d-hosts-file\") pod \"node-resolver-mjg8s\" (UID: \"0ed4f23e-1ad2-4346-8550-93565896543d\") " pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175063 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed4f23e-1ad2-4346-8550-93565896543d-tmp-dir\") pod \"node-resolver-mjg8s\" (UID: \"0ed4f23e-1ad2-4346-8550-93565896543d\") " pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175090 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8615e2e1-f518-4237-960f-2eda0b516165-serviceca\") pod \"node-ca-htq4b\" (UID: \"8615e2e1-f518-4237-960f-2eda0b516165\") " pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175115 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-socket-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175137 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-run\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175175 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-sysctl-d\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175210 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-host\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.175253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175240 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afb5107c-5103-4a50-8f07-b9325b1aba7d-tmp\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.175808 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175304 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.175808 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175363 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-registration-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.175808 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175394 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-device-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.175808 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175460 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkz7l\" (UniqueName: \"kubernetes.io/projected/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-kube-api-access-rkz7l\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.175808 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175485 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zjc\" (UniqueName: \"kubernetes.io/projected/afb5107c-5103-4a50-8f07-b9325b1aba7d-kube-api-access-b4zjc\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.175808 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175675 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 09:14:23.175808 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175675 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h6btv\"" Sep 10 09:14:23.176206 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175938 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 09:14:23.176206 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.175981 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 09:14:23.177068 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.176779 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bbnnn\"" Sep 10 09:14:23.177068 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.176979 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:23.177068 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.177042 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:23.179573 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.179485 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.182199 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.182169 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.182428 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.182348 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pxmgd\"" Sep 10 09:14:23.182850 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.182627 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 09:14:23.182850 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.182765 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 09:14:23.182975 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.182869 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 09:14:23.183035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.183017 2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 09:14:23.184669 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.184650 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 09:14:23.191045 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.191003 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 09:14:23.191319 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.191268 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 09:14:23.191781 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.191761 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 09:14:23.191781 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.191775 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5cr6z\"" Sep 10 09:14:23.192009 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.191991 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 09:14:23.192217 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.192200 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 09:14:23.220355 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.220331 2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5krm8" Sep 10 09:14:23.229914 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.229894 2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5krm8" Sep 10 09:14:23.263675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.263654 2546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 09:14:23.276744 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276717 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-system-cni-dir\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.276847 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276764 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-systemd-units\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.276847 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276794 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-run-ovn\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.276847 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276829 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afb5107c-5103-4a50-8f07-b9325b1aba7d-tmp\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.276968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276852 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f56cc9f-529f-4a84-a940-a9bef6f86110-cni-binary-copy\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.276968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276875 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-var-lib-kubelet\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.276968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276897 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8589e0d-ca1b-4278-9f30-1979024023f2-env-overrides\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.276968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276938 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.276968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276966 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-registration-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.277163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.276991 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-device-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.277163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277035 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkz7l\" (UniqueName: \"kubernetes.io/projected/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-kube-api-access-rkz7l\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.277163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277061 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-socket-dir-parent\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.277163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277065 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-registration-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.277163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277087 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-etc-kubernetes\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.277163 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.277090 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:23.277163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277121 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:23.277163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277136 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-device-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.277310 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret podName:ce6a92df-68a8-49fa-acb8-3b40e6852595 nodeName:}" failed. No retries permitted until 2026-09-10 09:14:23.777229547 +0000 UTC m=+2.958918595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret") pod "global-pull-secret-syncer-jr854" (UID: "ce6a92df-68a8-49fa-acb8-3b40e6852595") : object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277331 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-log-socket\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277356 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8615e2e1-f518-4237-960f-2eda0b516165-host\") pod \"node-ca-htq4b\" (UID: \"8615e2e1-f518-4237-960f-2eda0b516165\") " pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277372 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-sysctl-conf\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277393 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-hostroot\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277418 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-system-cni-dir\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277428 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8615e2e1-f518-4237-960f-2eda0b516165-host\") pod \"node-ca-htq4b\" (UID: \"8615e2e1-f518-4237-960f-2eda0b516165\") " pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277443 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277468 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-cni-bin\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277486 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-cni-netd\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277478 2546 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277504 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9m82\" (UniqueName: \"kubernetes.io/projected/b8589e0d-ca1b-4278-9f30-1979024023f2-kube-api-access-q9m82\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277507 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-sysctl-conf\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277522 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277566 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-var-lib-kubelet\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.277569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277574 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277597 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-daemon-config\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277656 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-run-multus-certs\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277682 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-var-lib-kubelet\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277716 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-var-lib-cni-multus\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277736 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4dt\" (UniqueName: \"kubernetes.io/projected/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-kube-api-access-bk4dt\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277753 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277796 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57b8bb57-7b18-4390-94cc-7a8e8a0c811e-agent-certs\") pod \"konnectivity-agent-j9qc8\" (UID: \"57b8bb57-7b18-4390-94cc-7a8e8a0c811e\") " pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277835 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6mw6\" (UniqueName: \"kubernetes.io/projected/0ed4f23e-1ad2-4346-8550-93565896543d-kube-api-access-k6mw6\") pod \"node-resolver-mjg8s\" (UID: \"0ed4f23e-1ad2-4346-8550-93565896543d\") " pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277862 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce6a92df-68a8-49fa-acb8-3b40e6852595-dbus\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277891 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-sys\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277957 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-sys\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.277998 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce6a92df-68a8-49fa-acb8-3b40e6852595-dbus\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278003 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-tuned\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278036 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhsm6\" (UniqueName: \"kubernetes.io/projected/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-kube-api-access-qhsm6\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278069 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-sys-fs\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278095 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-sysconfig\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.278401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278135 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-sys-fs\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278140 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-cnibin\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278175 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqd8c\" (UniqueName: \"kubernetes.io/projected/3f56cc9f-529f-4a84-a940-a9bef6f86110-kube-api-access-hqd8c\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278206 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-sysconfig\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278259 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-cni-binary-copy\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278290 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-kubelet\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278316 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-run-openvswitch\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278379 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8589e0d-ca1b-4278-9f30-1979024023f2-ovn-node-metrics-cert\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278409 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-sysctl-d\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278436 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-host\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278459 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-cni-dir\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278483 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-run-netns\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278511 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zjc\" (UniqueName: \"kubernetes.io/projected/afb5107c-5103-4a50-8f07-b9325b1aba7d-kube-api-access-b4zjc\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278535 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-sysctl-d\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278512 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-host\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278536 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-conf-dir\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278584 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.279090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278617 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wf57\" (UniqueName: \"kubernetes.io/projected/0edb23e0-a253-42a0-a1dd-6f17199ba610-kube-api-access-7wf57\") pod \"iptables-alerter-ctr4v\" (UID: \"0edb23e0-a253-42a0-a1dd-6f17199ba610\") " pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278654 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-slash\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278681 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-var-lib-openvswitch\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278708 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-kubernetes\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278741 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278773 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0edb23e0-a253-42a0-a1dd-6f17199ba610-host-slash\") pod \"iptables-alerter-ctr4v\" (UID: \"0edb23e0-a253-42a0-a1dd-6f17199ba610\") " pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278806 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8589e0d-ca1b-4278-9f30-1979024023f2-ovnkube-script-lib\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278837 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278839 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-kubernetes\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278863 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-lib-modules\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278898 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-run-k8s-cni-cncf-io\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278927 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57b8bb57-7b18-4390-94cc-7a8e8a0c811e-konnectivity-ca\") pod \"konnectivity-agent-j9qc8\" (UID: \"57b8bb57-7b18-4390-94cc-7a8e8a0c811e\") " pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278949 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278979 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-etc-openvswitch\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.278981 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-lib-modules\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279008 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-node-log\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279032 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.279642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279063 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6k2r\" (UniqueName: \"kubernetes.io/projected/8615e2e1-f518-4237-960f-2eda0b516165-kube-api-access-k6k2r\") pod \"node-ca-htq4b\" (UID: \"8615e2e1-f518-4237-960f-2eda0b516165\") " pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279090 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-modprobe-d\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279241 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-modprobe-d\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279249 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-var-lib-cni-bin\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279276 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0edb23e0-a253-42a0-a1dd-6f17199ba610-iptables-alerter-script\") pod \"iptables-alerter-ctr4v\" (UID: \"0edb23e0-a253-42a0-a1dd-6f17199ba610\") " pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279310 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-run-systemd\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279353 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8589e0d-ca1b-4278-9f30-1979024023f2-ovnkube-config\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279408 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-systemd\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279447 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-cnibin\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279475 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-run-netns\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279484 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-systemd\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279502 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279539 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce6a92df-68a8-49fa-acb8-3b40e6852595-kubelet-config\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279561 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ed4f23e-1ad2-4346-8550-93565896543d-hosts-file\") pod \"node-resolver-mjg8s\" (UID: \"0ed4f23e-1ad2-4346-8550-93565896543d\") " pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279583 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed4f23e-1ad2-4346-8550-93565896543d-tmp-dir\") pod \"node-resolver-mjg8s\" (UID: \"0ed4f23e-1ad2-4346-8550-93565896543d\") " pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279604 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8615e2e1-f518-4237-960f-2eda0b516165-serviceca\") pod \"node-ca-htq4b\" (UID: \"8615e2e1-f518-4237-960f-2eda0b516165\") " pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279626 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-socket-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.280152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279635 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce6a92df-68a8-49fa-acb8-3b40e6852595-kubelet-config\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.280674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279651 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-run\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.280674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279759 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afb5107c-5103-4a50-8f07-b9325b1aba7d-run\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.280674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.279948 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed4f23e-1ad2-4346-8550-93565896543d-tmp-dir\") pod \"node-resolver-mjg8s\" (UID: \"0ed4f23e-1ad2-4346-8550-93565896543d\") " pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.280674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.280045 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ed4f23e-1ad2-4346-8550-93565896543d-hosts-file\") pod \"node-resolver-mjg8s\" (UID: \"0ed4f23e-1ad2-4346-8550-93565896543d\") " pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.280674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.280169 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-socket-dir\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.280674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.280237 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8615e2e1-f518-4237-960f-2eda0b516165-serviceca\") pod \"node-ca-htq4b\" (UID: \"8615e2e1-f518-4237-960f-2eda0b516165\") " pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.281529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.281366 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afb5107c-5103-4a50-8f07-b9325b1aba7d-etc-tuned\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.281529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.281417 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afb5107c-5103-4a50-8f07-b9325b1aba7d-tmp\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.290112 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.290084 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zjc\" (UniqueName: \"kubernetes.io/projected/afb5107c-5103-4a50-8f07-b9325b1aba7d-kube-api-access-b4zjc\") pod \"tuned-dwbqv\" (UID: \"afb5107c-5103-4a50-8f07-b9325b1aba7d\") " pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.290260 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.290241 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6mw6\" (UniqueName: \"kubernetes.io/projected/0ed4f23e-1ad2-4346-8550-93565896543d-kube-api-access-k6mw6\") pod \"node-resolver-mjg8s\" (UID: \"0ed4f23e-1ad2-4346-8550-93565896543d\") " pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.290531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.290513 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkz7l\" (UniqueName: \"kubernetes.io/projected/64f62386-59d2-4fe5-994a-0f4c61c3a0ca-kube-api-access-rkz7l\") pod \"aws-ebs-csi-driver-node-wtt9j\" (UID: \"64f62386-59d2-4fe5-994a-0f4c61c3a0ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.290531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.290524 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6k2r\" (UniqueName: \"kubernetes.io/projected/8615e2e1-f518-4237-960f-2eda0b516165-kube-api-access-k6k2r\") pod \"node-ca-htq4b\" (UID: \"8615e2e1-f518-4237-960f-2eda0b516165\") " pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.379932 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.379904 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-cni-dir\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.380042 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.379949 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-run-netns\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380042 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.379981 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-conf-dir\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.380042 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380014 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.380209 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380082 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-conf-dir\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.380209 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380101 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-cni-dir\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.380209 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380115 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-run-netns\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380209 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380182 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wf57\" (UniqueName: \"kubernetes.io/projected/0edb23e0-a253-42a0-a1dd-6f17199ba610-kube-api-access-7wf57\") pod \"iptables-alerter-ctr4v\" (UID: \"0edb23e0-a253-42a0-a1dd-6f17199ba610\") " pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.380446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380211 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-slash\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380238 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-var-lib-openvswitch\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380277 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.380446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380317 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0edb23e0-a253-42a0-a1dd-6f17199ba610-host-slash\") pod \"iptables-alerter-ctr4v\" (UID: \"0edb23e0-a253-42a0-a1dd-6f17199ba610\") " pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.380446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380342 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8589e0d-ca1b-4278-9f30-1979024023f2-ovnkube-script-lib\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380391 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-run-k8s-cni-cncf-io\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.380446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380405 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-var-lib-openvswitch\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380418 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57b8bb57-7b18-4390-94cc-7a8e8a0c811e-konnectivity-ca\") pod \"konnectivity-agent-j9qc8\" (UID: \"57b8bb57-7b18-4390-94cc-7a8e8a0c811e\") " pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:23.380840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380443 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-etc-openvswitch\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380479 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-node-log\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380505 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380572 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-var-lib-cni-bin\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.380840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380614 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0edb23e0-a253-42a0-a1dd-6f17199ba610-iptables-alerter-script\") pod \"iptables-alerter-ctr4v\" (UID: \"0edb23e0-a253-42a0-a1dd-6f17199ba610\") " pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.380840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380641 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-run-systemd\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380659 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-slash\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380667 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8589e0d-ca1b-4278-9f30-1979024023f2-ovnkube-config\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.380840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.380708 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-etc-openvswitch\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.381270 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381083 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-cnibin\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.381270 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381098 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-cnibin\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.381270 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381160 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-run-netns\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.381270 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381197 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.381270 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381214 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-var-lib-cni-bin\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.381537 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381288 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-system-cni-dir\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.381537 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381458 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.381643 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381534 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0edb23e0-a253-42a0-a1dd-6f17199ba610-host-slash\") pod \"iptables-alerter-ctr4v\" (UID: \"0edb23e0-a253-42a0-a1dd-6f17199ba610\") " pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.381643 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381587 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-systemd-units\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.381748 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381654 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b8589e0d-ca1b-4278-9f30-1979024023f2-ovnkube-config\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.381800 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381756 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-node-log\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.381895 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381864 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-run-netns\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.382139 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382115 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57b8bb57-7b18-4390-94cc-7a8e8a0c811e-konnectivity-ca\") pod \"konnectivity-agent-j9qc8\" (UID: \"57b8bb57-7b18-4390-94cc-7a8e8a0c811e\") " pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:23.382247 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382228 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.382312 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382298 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.382427 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382394 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-run-systemd\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.382542 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.381494 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-systemd-units\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.382542 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382531 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0edb23e0-a253-42a0-a1dd-6f17199ba610-iptables-alerter-script\") pod \"iptables-alerter-ctr4v\" (UID: \"0edb23e0-a253-42a0-a1dd-6f17199ba610\") " pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.382709 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382689 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-system-cni-dir\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.382776 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382760 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-run-ovn\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.382832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382820 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-run-k8s-cni-cncf-io\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.382884 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382861 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.382884 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382872 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-run-ovn\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.382980 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.382919 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f56cc9f-529f-4a84-a940-a9bef6f86110-cni-binary-copy\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.383028 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383006 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-var-lib-kubelet\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.383079 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383037 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8589e0d-ca1b-4278-9f30-1979024023f2-env-overrides\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.383131 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383083 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-socket-dir-parent\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.383131 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383112 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-etc-kubernetes\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.383222 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383142 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:23.383222 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383199 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-log-socket\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.383318 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383247 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-hostroot\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.383318 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383295 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-socket-dir-parent\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.383447 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383292 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-system-cni-dir\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.383447 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383331 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b8589e0d-ca1b-4278-9f30-1979024023f2-ovnkube-script-lib\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.383447 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383425 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-system-cni-dir\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.383583 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.383466 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:23.383583 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383476 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-etc-kubernetes\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.383583 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383532 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-hostroot\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.383583 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383576 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-var-lib-kubelet\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.383762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383636 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-log-socket\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.383762 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.383537 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs podName:ea243c6d-7906-4f8f-8da3-5e1d888d4d6a nodeName:}" failed. No retries permitted until 2026-09-10 09:14:23.883518147 +0000 UTC m=+3.065207211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs") pod "network-metrics-daemon-knj5j" (UID: "ea243c6d-7906-4f8f-8da3-5e1d888d4d6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:23.383762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383684 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:23.383949 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383787 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-cni-bin\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.383949 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383815 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-cni-netd\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.383949 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383862 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9m82\" (UniqueName: \"kubernetes.io/projected/b8589e0d-ca1b-4278-9f30-1979024023f2-kube-api-access-q9m82\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.383949 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383917 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-daemon-config\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.384143 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383949 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-run-multus-certs\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.384143 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.383988 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f56cc9f-529f-4a84-a940-a9bef6f86110-cni-binary-copy\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.384143 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384011 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-var-lib-cni-multus\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.384143 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384028 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-cni-bin\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.384143 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384047 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4dt\" (UniqueName: \"kubernetes.io/projected/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-kube-api-access-bk4dt\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:23.384143 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384086 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-run-multus-certs\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.384143 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384097 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.384486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384155 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57b8bb57-7b18-4390-94cc-7a8e8a0c811e-agent-certs\") pod \"konnectivity-agent-j9qc8\" (UID: \"57b8bb57-7b18-4390-94cc-7a8e8a0c811e\") " pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:23.384486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384204 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhsm6\" (UniqueName: \"kubernetes.io/projected/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-kube-api-access-qhsm6\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.384486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384241 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-cnibin\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.384486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384277 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqd8c\" (UniqueName: \"kubernetes.io/projected/3f56cc9f-529f-4a84-a940-a9bef6f86110-kube-api-access-hqd8c\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.384486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384310 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-cni-binary-copy\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.384486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384344 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-kubelet\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.384486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384391 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-run-openvswitch\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.384486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384423 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8589e0d-ca1b-4278-9f30-1979024023f2-ovn-node-metrics-cert\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.384486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384442 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b8589e0d-ca1b-4278-9f30-1979024023f2-env-overrides\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.384926 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384636 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.384926 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384750 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-cni-netd\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.384926 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384822 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3f56cc9f-529f-4a84-a940-a9bef6f86110-host-var-lib-cni-multus\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.385063 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.384943 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3f56cc9f-529f-4a84-a940-a9bef6f86110-multus-daemon-config\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.385063 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.385000 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-cnibin\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.385160 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.385125 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-run-openvswitch\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.385223 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.385184 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8589e0d-ca1b-4278-9f30-1979024023f2-host-kubelet\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.386494 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.386465 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-cni-binary-copy\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.387793 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.387766 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8589e0d-ca1b-4278-9f30-1979024023f2-ovn-node-metrics-cert\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.388057 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.388028 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57b8bb57-7b18-4390-94cc-7a8e8a0c811e-agent-certs\") pod \"konnectivity-agent-j9qc8\" (UID: \"57b8bb57-7b18-4390-94cc-7a8e8a0c811e\") " pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:23.389579 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.389558 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wf57\" (UniqueName: \"kubernetes.io/projected/0edb23e0-a253-42a0-a1dd-6f17199ba610-kube-api-access-7wf57\") pod \"iptables-alerter-ctr4v\" (UID: \"0edb23e0-a253-42a0-a1dd-6f17199ba610\") " pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.390474 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.390455 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 09:14:23.390474 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.390477 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 09:14:23.390604 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.390488 2546 projected.go:196] Error preparing data for projected volume kube-api-access-k28q6 for pod openshift-network-diagnostics/network-check-target-b49hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:23.390604 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.390553 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6 podName:52423870-4667-4de2-b4ee-6418b9edefaf nodeName:}" failed. No retries permitted until 2026-09-10 09:14:23.890535829 +0000 UTC m=+3.072224877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k28q6" (UniqueName: "kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6") pod "network-check-target-b49hb" (UID: "52423870-4667-4de2-b4ee-6418b9edefaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:23.394034 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.394008 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9m82\" (UniqueName: \"kubernetes.io/projected/b8589e0d-ca1b-4278-9f30-1979024023f2-kube-api-access-q9m82\") pod \"ovnkube-node-xsql9\" (UID: \"b8589e0d-ca1b-4278-9f30-1979024023f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.394579 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.394507 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4dt\" (UniqueName: \"kubernetes.io/projected/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-kube-api-access-bk4dt\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:23.394675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.394602 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqd8c\" (UniqueName: \"kubernetes.io/projected/3f56cc9f-529f-4a84-a940-a9bef6f86110-kube-api-access-hqd8c\") pod \"multus-zwqd9\" (UID: \"3f56cc9f-529f-4a84-a940-a9bef6f86110\") " pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.394950 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.394927 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhsm6\" (UniqueName: \"kubernetes.io/projected/bcdf75da-e1be-4a9d-a514-fbd4e04e4766-kube-api-access-qhsm6\") pod \"multus-additional-cni-plugins-5tt2m\" (UID: \"bcdf75da-e1be-4a9d-a514-fbd4e04e4766\") " pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.476545 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.476490 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" Sep 10 09:14:23.491400 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.491379 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-htq4b" Sep 10 09:14:23.495207 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.495192 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" Sep 10 09:14:23.503123 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.503107 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mjg8s" Sep 10 09:14:23.512966 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.512944 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" Sep 10 09:14:23.521713 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.521697 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zwqd9" Sep 10 09:14:23.529455 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.529437 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:23.537098 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.537075 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctr4v" Sep 10 09:14:23.541952 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.541931 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:23.659740 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.659715 2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 09:14:23.787882 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.787796 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:23.788034 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.787965 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:23.788034 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.788035 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret podName:ce6a92df-68a8-49fa-acb8-3b40e6852595 nodeName:}" failed. No retries permitted until 2026-09-10 09:14:24.788013916 +0000 UTC m=+3.969702976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret") pod "global-pull-secret-syncer-jr854" (UID: "ce6a92df-68a8-49fa-acb8-3b40e6852595") : object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:23.888525 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.888495 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:23.888686 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.888662 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:23.888830 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.888734 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs podName:ea243c6d-7906-4f8f-8da3-5e1d888d4d6a nodeName:}" failed. No retries permitted until 2026-09-10 09:14:24.888718276 +0000 UTC m=+4.070407339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs") pod "network-metrics-daemon-knj5j" (UID: "ea243c6d-7906-4f8f-8da3-5e1d888d4d6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:23.989081 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:23.989051 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:23.989302 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.989263 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 09:14:23.989427 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.989312 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 09:14:23.989427 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.989328 2546 projected.go:196] Error preparing data for projected volume kube-api-access-k28q6 for pod openshift-network-diagnostics/network-check-target-b49hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:23.989427 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:23.989393 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6 podName:52423870-4667-4de2-b4ee-6418b9edefaf nodeName:}" failed. No retries permitted until 2026-09-10 09:14:24.989372036 +0000 UTC m=+4.171061127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k28q6" (UniqueName: "kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6") pod "network-check-target-b49hb" (UID: "52423870-4667-4de2-b4ee-6418b9edefaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:24.233561 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:24.233468 2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 09:09:23 +0000 UTC" deadline="2028-03-03 19:18:49.947134847 +0000 UTC" Sep 10 09:14:24.233561 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:24.233507 2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12970h4m25.713632739s" Sep 10 09:14:24.340532 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:24.340499 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f62386_59d2_4fe5_994a_0f4c61c3a0ca.slice/crio-a28dd4414e28914ae58207129fecadf553c2f57409231b09466de5260c0fe38a WatchSource:0}: Error finding container a28dd4414e28914ae58207129fecadf553c2f57409231b09466de5260c0fe38a: Status 404 returned error can't find the container with id a28dd4414e28914ae58207129fecadf553c2f57409231b09466de5260c0fe38a Sep 10 09:14:24.341847 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:24.341823 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8589e0d_ca1b_4278_9f30_1979024023f2.slice/crio-41ba191b50317ea908db217d42b714caaea882fbaf5047a9d75ab5dad2a72c9a WatchSource:0}: Error finding container 41ba191b50317ea908db217d42b714caaea882fbaf5047a9d75ab5dad2a72c9a: Status 404 returned error can't find the container with id 41ba191b50317ea908db217d42b714caaea882fbaf5047a9d75ab5dad2a72c9a Sep 10 09:14:24.343817 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:24.343257 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed4f23e_1ad2_4346_8550_93565896543d.slice/crio-14a8abca391f2a31eb2e8be4e8927564b138e571985d48ba85cb0d88be4bea00 WatchSource:0}: Error finding container 14a8abca391f2a31eb2e8be4e8927564b138e571985d48ba85cb0d88be4bea00: Status 404 returned error can't find the container with id 14a8abca391f2a31eb2e8be4e8927564b138e571985d48ba85cb0d88be4bea00 Sep 10 09:14:24.346277 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:24.346197 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb5107c_5103_4a50_8f07_b9325b1aba7d.slice/crio-76886c166fbb23197ad97e55888705782ef222777631c6bb8918ad0cac019bb5 WatchSource:0}: Error finding container 76886c166fbb23197ad97e55888705782ef222777631c6bb8918ad0cac019bb5: Status 404 returned error can't find the container with id 76886c166fbb23197ad97e55888705782ef222777631c6bb8918ad0cac019bb5 Sep 10 09:14:24.347817 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:24.347700 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b8bb57_7b18_4390_94cc_7a8e8a0c811e.slice/crio-d9dd21a9ebf51195478f718d6f3742f151ed5e8cade095bf0a9b2acad1a5aa75 WatchSource:0}: Error finding container d9dd21a9ebf51195478f718d6f3742f151ed5e8cade095bf0a9b2acad1a5aa75: Status 404 returned error can't find the container with id d9dd21a9ebf51195478f718d6f3742f151ed5e8cade095bf0a9b2acad1a5aa75 Sep 10 09:14:24.348310 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:24.348289 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f56cc9f_529f_4a84_a940_a9bef6f86110.slice/crio-40054565b1a5609d63f0424eda75a28c752dfeb1f12b406d19732326d82e8cec WatchSource:0}: Error finding container 40054565b1a5609d63f0424eda75a28c752dfeb1f12b406d19732326d82e8cec: Status 404 returned error can't find the container with id 40054565b1a5609d63f0424eda75a28c752dfeb1f12b406d19732326d82e8cec Sep 10 09:14:24.349370 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:14:24.349344 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdf75da_e1be_4a9d_a514_fbd4e04e4766.slice/crio-bf0cb85e79ddce0bdab3ff23df51604d42193915aa9fd7bb250ea5e92bc5a0ac WatchSource:0}: Error finding container bf0cb85e79ddce0bdab3ff23df51604d42193915aa9fd7bb250ea5e92bc5a0ac: Status 404 returned error can't find the container with id bf0cb85e79ddce0bdab3ff23df51604d42193915aa9fd7bb250ea5e92bc5a0ac Sep 10 09:14:24.795877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:24.795841 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:24.796034 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:24.796016 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:24.796129 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:24.796088 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret podName:ce6a92df-68a8-49fa-acb8-3b40e6852595 nodeName:}" failed. No retries permitted until 2026-09-10 09:14:26.796061185 +0000 UTC m=+5.977750239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret") pod "global-pull-secret-syncer-jr854" (UID: "ce6a92df-68a8-49fa-acb8-3b40e6852595") : object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:24.896914 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:24.896834 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:24.897062 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:24.896993 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:24.897062 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:24.897052 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs podName:ea243c6d-7906-4f8f-8da3-5e1d888d4d6a nodeName:}" failed. No retries permitted until 2026-09-10 09:14:26.897034097 +0000 UTC m=+6.078723147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs") pod "network-metrics-daemon-knj5j" (UID: "ea243c6d-7906-4f8f-8da3-5e1d888d4d6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:24.997821 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:24.997779 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:24.997980 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:24.997939 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 09:14:24.997980 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:24.997962 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 09:14:24.997980 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:24.997975 2546 projected.go:196] Error preparing data for projected volume kube-api-access-k28q6 for pod openshift-network-diagnostics/network-check-target-b49hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:24.998161 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:24.998037 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6 podName:52423870-4667-4de2-b4ee-6418b9edefaf nodeName:}" failed. No retries permitted until 2026-09-10 09:14:26.998017283 +0000 UTC m=+6.179706345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k28q6" (UniqueName: "kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6") pod "network-check-target-b49hb" (UID: "52423870-4667-4de2-b4ee-6418b9edefaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:25.234184 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.234083 2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 09:09:23 +0000 UTC" deadline="2028-05-08 11:47:57.91992605 +0000 UTC" Sep 10 09:14:25.234184 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.234123 2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14546h33m32.685807751s" Sep 10 09:14:25.257950 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.257913 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:25.258104 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:25.258042 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:25.258511 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.258490 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:25.258627 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:25.258610 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:25.258700 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.258690 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:25.258831 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:25.258784 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:25.268314 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.267743 2546 generic.go:356] "Generic (PLEG): container finished" podID="050d88a1f009d75cf4482ef0a62462f4" containerID="af3ed52137e5c08382bb80d0e0de3ee90d72079322ab3eb81ffbb9b97a1fb2ca" exitCode=0 Sep 10 09:14:25.268314 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.267867 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" event={"ID":"050d88a1f009d75cf4482ef0a62462f4","Type":"ContainerDied","Data":"af3ed52137e5c08382bb80d0e0de3ee90d72079322ab3eb81ffbb9b97a1fb2ca"} Sep 10 09:14:25.272674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.272645 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" event={"ID":"a131841e4a19fc7a8712e64d2e2b1c9d","Type":"ContainerStarted","Data":"a7f27f8f6fa882079088de3eed6b10bdfa5d9edfa4d9a10ce497d93c78d8c239"} Sep 10 09:14:25.294645 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.294616 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-htq4b" event={"ID":"8615e2e1-f518-4237-960f-2eda0b516165","Type":"ContainerStarted","Data":"0b8a30ed125240f6714c1b1eb6a2be3b81aab19e9340b74d7797bcc30ff1d051"} Sep 10 09:14:25.302217 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.302188 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" event={"ID":"bcdf75da-e1be-4a9d-a514-fbd4e04e4766","Type":"ContainerStarted","Data":"bf0cb85e79ddce0bdab3ff23df51604d42193915aa9fd7bb250ea5e92bc5a0ac"} Sep 10 09:14:25.307681 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.307655 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zwqd9" event={"ID":"3f56cc9f-529f-4a84-a940-a9bef6f86110","Type":"ContainerStarted","Data":"40054565b1a5609d63f0424eda75a28c752dfeb1f12b406d19732326d82e8cec"} Sep 10 09:14:25.316127 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.316102 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" event={"ID":"afb5107c-5103-4a50-8f07-b9325b1aba7d","Type":"ContainerStarted","Data":"76886c166fbb23197ad97e55888705782ef222777631c6bb8918ad0cac019bb5"} Sep 10 09:14:25.341698 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.341638 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" event={"ID":"64f62386-59d2-4fe5-994a-0f4c61c3a0ca","Type":"ContainerStarted","Data":"a28dd4414e28914ae58207129fecadf553c2f57409231b09466de5260c0fe38a"} Sep 10 09:14:25.357984 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.357913 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctr4v" event={"ID":"0edb23e0-a253-42a0-a1dd-6f17199ba610","Type":"ContainerStarted","Data":"0d31ba9ad0d9b018387e02c13a8a540011e6be33005bd247cd0da77abe92b801"} Sep 10 09:14:25.359947 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.359866 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j9qc8" event={"ID":"57b8bb57-7b18-4390-94cc-7a8e8a0c811e","Type":"ContainerStarted","Data":"d9dd21a9ebf51195478f718d6f3742f151ed5e8cade095bf0a9b2acad1a5aa75"} Sep 10 09:14:25.371024 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.370934 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mjg8s" event={"ID":"0ed4f23e-1ad2-4346-8550-93565896543d","Type":"ContainerStarted","Data":"14a8abca391f2a31eb2e8be4e8927564b138e571985d48ba85cb0d88be4bea00"} Sep 10 09:14:25.378481 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:25.378454 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" event={"ID":"b8589e0d-ca1b-4278-9f30-1979024023f2","Type":"ContainerStarted","Data":"41ba191b50317ea908db217d42b714caaea882fbaf5047a9d75ab5dad2a72c9a"} Sep 10 09:14:26.402678 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:26.402413 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" event={"ID":"050d88a1f009d75cf4482ef0a62462f4","Type":"ContainerStarted","Data":"df7f5590377134c17cc7675eda71f4da26d594d561de61caef7a7ba4387ff12e"} Sep 10 09:14:26.422539 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:26.422473 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-170.ec2.internal" podStartSLOduration=4.422451192 podStartE2EDuration="4.422451192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:25.309258769 +0000 UTC m=+4.490947858" watchObservedRunningTime="2026-09-10 09:14:26.422451192 +0000 UTC m=+5.604140264" Sep 10 09:14:26.422705 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:26.422677 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-170.ec2.internal" podStartSLOduration=4.422669076 podStartE2EDuration="4.422669076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:26.421549615 +0000 UTC m=+5.603238704" watchObservedRunningTime="2026-09-10 09:14:26.422669076 +0000 UTC m=+5.604358148" Sep 10 09:14:26.815686 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:26.815601 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:26.815844 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:26.815771 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:26.815910 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:26.815847 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret podName:ce6a92df-68a8-49fa-acb8-3b40e6852595 nodeName:}" failed. No retries permitted until 2026-09-10 09:14:30.815825823 +0000 UTC m=+9.997514883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret") pod "global-pull-secret-syncer-jr854" (UID: "ce6a92df-68a8-49fa-acb8-3b40e6852595") : object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:26.916633 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:26.916520 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:26.916796 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:26.916699 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:26.916796 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:26.916760 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs podName:ea243c6d-7906-4f8f-8da3-5e1d888d4d6a nodeName:}" failed. No retries permitted until 2026-09-10 09:14:30.916740564 +0000 UTC m=+10.098429615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs") pod "network-metrics-daemon-knj5j" (UID: "ea243c6d-7906-4f8f-8da3-5e1d888d4d6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:27.017424 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:27.017381 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:27.017584 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:27.017551 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 09:14:27.017584 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:27.017573 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 09:14:27.017731 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:27.017587 2546 projected.go:196] Error preparing data for projected volume kube-api-access-k28q6 for pod openshift-network-diagnostics/network-check-target-b49hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:27.017731 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:27.017648 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6 podName:52423870-4667-4de2-b4ee-6418b9edefaf nodeName:}" failed. No retries permitted until 2026-09-10 09:14:31.017629604 +0000 UTC m=+10.199318655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k28q6" (UniqueName: "kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6") pod "network-check-target-b49hb" (UID: "52423870-4667-4de2-b4ee-6418b9edefaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:27.255512 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:27.255430 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:27.255676 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:27.255571 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:27.255947 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:27.255920 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:27.256063 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:27.256040 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:27.256129 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:27.256107 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:27.256203 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:27.256184 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:29.256362 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:29.256329 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:29.256871 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:29.256474 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:29.256871 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:29.256477 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:29.256871 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:29.256545 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:29.256871 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:29.256654 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:29.256871 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:29.256747 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:30.851750 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:30.851715 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:30.852215 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:30.851863 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:30.852215 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:30.851939 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret podName:ce6a92df-68a8-49fa-acb8-3b40e6852595 nodeName:}" failed. No retries permitted until 2026-09-10 09:14:38.85191707 +0000 UTC m=+18.033606139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret") pod "global-pull-secret-syncer-jr854" (UID: "ce6a92df-68a8-49fa-acb8-3b40e6852595") : object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:30.952346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:30.952311 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:30.952516 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:30.952486 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:30.952566 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:30.952560 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs podName:ea243c6d-7906-4f8f-8da3-5e1d888d4d6a nodeName:}" failed. No retries permitted until 2026-09-10 09:14:38.952540617 +0000 UTC m=+18.134229674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs") pod "network-metrics-daemon-knj5j" (UID: "ea243c6d-7906-4f8f-8da3-5e1d888d4d6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:31.053416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:31.053333 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:31.053567 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:31.053486 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 09:14:31.053567 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:31.053518 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 09:14:31.053567 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:31.053531 2546 projected.go:196] Error preparing data for projected volume kube-api-access-k28q6 for pod openshift-network-diagnostics/network-check-target-b49hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:31.053746 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:31.053602 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6 podName:52423870-4667-4de2-b4ee-6418b9edefaf nodeName:}" failed. No retries permitted until 2026-09-10 09:14:39.053570416 +0000 UTC m=+18.235259473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k28q6" (UniqueName: "kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6") pod "network-check-target-b49hb" (UID: "52423870-4667-4de2-b4ee-6418b9edefaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:31.257340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:31.256962 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:31.257340 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:31.257081 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:31.257599 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:31.257462 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:31.257599 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:31.257488 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:31.257599 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:31.257563 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:31.258600 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:31.258547 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:33.256088 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:33.256028 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:33.256088 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:33.256078 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:33.256633 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:33.256148 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:33.256633 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:33.256169 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:33.256633 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:33.256236 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:33.256633 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:33.256360 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:35.255640 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:35.255607 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:35.255640 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:35.255623 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:35.255993 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:35.255719 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:35.255993 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:35.255767 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:35.255993 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:35.255844 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:35.255993 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:35.255902 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:37.255228 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:37.255194 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:37.255694 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:37.255198 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:37.255694 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:37.255328 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:37.255694 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:37.255409 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:37.255694 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:37.255480 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:37.255694 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:37.255570 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:38.914738 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:38.914652 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:38.915170 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:38.914805 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:38.915170 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:38.914871 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret podName:ce6a92df-68a8-49fa-acb8-3b40e6852595 nodeName:}" failed. No retries permitted until 2026-09-10 09:14:54.914857126 +0000 UTC m=+34.096546183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret") pod "global-pull-secret-syncer-jr854" (UID: "ce6a92df-68a8-49fa-acb8-3b40e6852595") : object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:39.015674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:39.015643 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:39.015833 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:39.015791 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:39.015890 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:39.015868 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs podName:ea243c6d-7906-4f8f-8da3-5e1d888d4d6a nodeName:}" failed. No retries permitted until 2026-09-10 09:14:55.015845829 +0000 UTC m=+34.197534876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs") pod "network-metrics-daemon-knj5j" (UID: "ea243c6d-7906-4f8f-8da3-5e1d888d4d6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:39.116835 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:39.116804 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:39.116993 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:39.116977 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 09:14:39.117044 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:39.117005 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 09:14:39.117044 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:39.117016 2546 projected.go:196] Error preparing data for projected volume kube-api-access-k28q6 for pod openshift-network-diagnostics/network-check-target-b49hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:39.117115 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:39.117079 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6 podName:52423870-4667-4de2-b4ee-6418b9edefaf nodeName:}" failed. No retries permitted until 2026-09-10 09:14:55.117065327 +0000 UTC m=+34.298754371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k28q6" (UniqueName: "kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6") pod "network-check-target-b49hb" (UID: "52423870-4667-4de2-b4ee-6418b9edefaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:39.255461 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:39.255434 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:39.255630 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:39.255549 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:39.255630 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:39.255558 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:39.255744 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:39.255650 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:39.255744 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:39.255675 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:39.255744 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:39.255739 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:41.258414 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:41.258384 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:41.258414 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:41.258412 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:41.258889 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:41.258384 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:41.258889 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:41.258483 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:41.258889 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:41.258555 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:41.258889 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:41.258629 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:43.255862 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.255702 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:43.256485 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.255703 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:43.256485 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:43.255941 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:43.256485 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.255703 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:43.256485 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:43.256004 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:43.256485 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:43.256076 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:43.441468 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.441439 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-htq4b" event={"ID":"8615e2e1-f518-4237-960f-2eda0b516165","Type":"ContainerStarted","Data":"98a38d12075aa1ecffbba4b78408ed80f65168dec14b39547d150c565ac11860"} Sep 10 09:14:43.442906 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.442884 2546 generic.go:356] "Generic (PLEG): container finished" podID="bcdf75da-e1be-4a9d-a514-fbd4e04e4766" containerID="e4b32f52aa544a025cf0b8fdffd503e8d45f74a6c4dc053dd9aa856a551e99db" exitCode=0 Sep 10 09:14:43.443000 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.442959 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" event={"ID":"bcdf75da-e1be-4a9d-a514-fbd4e04e4766","Type":"ContainerDied","Data":"e4b32f52aa544a025cf0b8fdffd503e8d45f74a6c4dc053dd9aa856a551e99db"} Sep 10 09:14:43.444642 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.444620 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zwqd9" event={"ID":"3f56cc9f-529f-4a84-a940-a9bef6f86110","Type":"ContainerStarted","Data":"a330884e69f7660eb5e59ef30f6821c1ab93e5a4cc5e4d78ca2c6c0f634dcc93"} Sep 10 09:14:43.446263 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.446241 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" event={"ID":"afb5107c-5103-4a50-8f07-b9325b1aba7d","Type":"ContainerStarted","Data":"3e6d4413b58e9db4bd5440d4c02ac77e335367d569d148c966c3458d8618af57"} Sep 10 09:14:43.447728 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.447701 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" event={"ID":"64f62386-59d2-4fe5-994a-0f4c61c3a0ca","Type":"ContainerStarted","Data":"ec331f6b5a03ce07c08906c8e61b5b10e431154bc8e8d353e4c664d6fc68e8e7"} Sep 10 09:14:43.449128 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.449099 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j9qc8" event={"ID":"57b8bb57-7b18-4390-94cc-7a8e8a0c811e","Type":"ContainerStarted","Data":"44d7c507e0bbbc19ac7939df5acbed46a05e49b59d06a11201102b43afb3bd35"} Sep 10 09:14:43.450504 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.450484 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mjg8s" event={"ID":"0ed4f23e-1ad2-4346-8550-93565896543d","Type":"ContainerStarted","Data":"e99025b3aab1398464fae82d083daf762a728e4a0cf262d90682811856d1c0c5"} Sep 10 09:14:43.453375 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.453354 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" event={"ID":"b8589e0d-ca1b-4278-9f30-1979024023f2","Type":"ContainerStarted","Data":"e9dcf9185c0d496263f10930e3bae71ed980ad302ed97b6d8bbd5073e7444a1a"} Sep 10 09:14:43.453455 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.453407 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" event={"ID":"b8589e0d-ca1b-4278-9f30-1979024023f2","Type":"ContainerStarted","Data":"8ba1d672069003f6939b39ef64a78cea882458cf21df6e606bf6d337ce898045"} Sep 10 09:14:43.453455 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.453438 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" event={"ID":"b8589e0d-ca1b-4278-9f30-1979024023f2","Type":"ContainerStarted","Data":"f3281873c727730fcddab84debb60a0f0bb6723c55f549d7c6e72e64150308db"} Sep 10 09:14:43.453535 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.453465 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" event={"ID":"b8589e0d-ca1b-4278-9f30-1979024023f2","Type":"ContainerStarted","Data":"8d3177b8ad8f42c97a9997366a243ed32658e66fe0c1770c588634b62eb93b6c"} Sep 10 09:14:43.453535 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.453525 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" event={"ID":"b8589e0d-ca1b-4278-9f30-1979024023f2","Type":"ContainerStarted","Data":"0e24ebc054357cf73f31ae90f6eb7a7693a9cae75740ba083c2688cf3c896048"} Sep 10 09:14:43.453602 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.453552 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" event={"ID":"b8589e0d-ca1b-4278-9f30-1979024023f2","Type":"ContainerStarted","Data":"4262949d28da6e0b3ba09f4587e16b78364520e9c4f4a3e09c129085c2b34a7e"} Sep 10 09:14:43.454980 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.454932 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-htq4b" podStartSLOduration=3.370654332 podStartE2EDuration="21.454919139s" totalImagesPullingTime="18.084264807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:43.454862688 +0000 UTC m=+22.636551771" watchObservedRunningTime="2026-09-10 09:14:43.454919139 +0000 UTC m=+22.636608205" Sep 10 09:14:43.472000 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.471958 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dwbqv" podStartSLOduration=4.363753169 podStartE2EDuration="22.471947859s" totalImagesPullingTime="18.10819469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:43.471691865 +0000 UTC m=+22.653380932" watchObservedRunningTime="2026-09-10 09:14:43.471947859 +0000 UTC m=+22.653636925" Sep 10 09:14:43.485433 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.485400 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mjg8s" podStartSLOduration=3.376022022 podStartE2EDuration="21.485389361s" totalImagesPullingTime="18.109367339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:43.485068625 +0000 UTC m=+22.666757690" watchObservedRunningTime="2026-09-10 09:14:43.485389361 +0000 UTC m=+22.667078424" Sep 10 09:14:43.502206 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.502173 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zwqd9" podStartSLOduration=3.348774749 podStartE2EDuration="21.50216318s" totalImagesPullingTime="18.153388431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:43.502005635 +0000 UTC m=+22.683694701" watchObservedRunningTime="2026-09-10 09:14:43.50216318 +0000 UTC m=+22.683852245" Sep 10 09:14:43.544274 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.544230 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j9qc8" podStartSLOduration=7.257626417 podStartE2EDuration="21.544214703s" totalImagesPullingTime="14.286588286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:43.520897314 +0000 UTC m=+22.702586390" watchObservedRunningTime="2026-09-10 09:14:43.544214703 +0000 UTC m=+22.725903789" Sep 10 09:14:43.598793 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.598766 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:43.681984 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.681961 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:43.778063 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.778033 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:43.837469 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.837400 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:43.962101 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:43.962070 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:44.020486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:44.020462 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:44.320856 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:44.320835 2546 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 09:14:44.457670 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:44.457602 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" event={"ID":"64f62386-59d2-4fe5-994a-0f4c61c3a0ca","Type":"ContainerStarted","Data":"b02e8bc2127bdf7a45069647e05375b986039e4a814d677253b61026e41e8bc2"} Sep 10 09:14:44.459222 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:44.459190 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctr4v" event={"ID":"0edb23e0-a253-42a0-a1dd-6f17199ba610","Type":"ContainerStarted","Data":"f4ded8dd365ad65a0ebd581169092ec6d32ca25b45591e320537e317616a040c"} Sep 10 09:14:44.473186 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:44.473134 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ctr4v" podStartSLOduration=8.190260627 podStartE2EDuration="22.47311542s" totalImagesPullingTime="14.282854793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:44.473069179 +0000 UTC m=+23.654758265" watchObservedRunningTime="2026-09-10 09:14:44.47311542 +0000 UTC m=+23.654804488" Sep 10 09:14:44.762162 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:44.762126 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:45.054471 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.054392 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:45.055030 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.055007 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:45.157547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.157517 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:45.225422 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.225328 2546 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T09:14:44.320852197Z","UUID":"45040fd8-3b62-41db-b00b-16a8e276883c","Handler":null,"Name":"","Endpoint":""} Sep 10 09:14:45.230538 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.227885 2546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 09:14:45.230538 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.227932 2546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 09:14:45.240889 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.240862 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:45.256352 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.256256 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:45.256352 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.256298 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:45.256352 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.256298 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:45.256542 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:45.256398 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:45.256542 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:45.256476 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:45.256664 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:45.256540 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:45.315345 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.315261 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:45.462719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.462691 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" event={"ID":"64f62386-59d2-4fe5-994a-0f4c61c3a0ca","Type":"ContainerStarted","Data":"2fe394e48a84c07a9b559d731df4dc4419b10c0eb425fe630db0afc25a0eebd3"} Sep 10 09:14:45.465819 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.465796 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" event={"ID":"b8589e0d-ca1b-4278-9f30-1979024023f2","Type":"ContainerStarted","Data":"b9c38e70fa299ca9fb5d618a68ac7e8facbe0c99020adfc6bf1c723a1292acda"} Sep 10 09:14:45.466102 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.466083 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:45.466469 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.466453 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j9qc8" Sep 10 09:14:45.485544 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.485507 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wtt9j" podStartSLOduration=3.9725790610000002 podStartE2EDuration="24.485494434s" totalImagesPullingTime="20.512915373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:45.485042977 +0000 UTC m=+24.666732062" watchObservedRunningTime="2026-09-10 09:14:45.485494434 +0000 UTC m=+24.667183500" Sep 10 09:14:45.562513 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:45.562492 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:46.137646 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:46.137613 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:46.196851 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:46.196817 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:46.581863 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:46.581827 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:47.255615 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:47.255584 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:47.255801 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:47.255701 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:47.255801 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:47.255589 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:47.255911 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:47.255817 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:47.255911 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:47.255853 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:47.256021 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:47.255922 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:48.475662 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:48.475438 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" event={"ID":"bcdf75da-e1be-4a9d-a514-fbd4e04e4766","Type":"ContainerStarted","Data":"5bcab6371f1b95a1e0a82510db3bf76fe4af2590c02e0524e905f1215bccb59d"} Sep 10 09:14:48.479243 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:48.479220 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" event={"ID":"b8589e0d-ca1b-4278-9f30-1979024023f2","Type":"ContainerStarted","Data":"561543e02c43d85c7dfa95f55d2a0922cc7aa59c1bffd07e0c5651ab3d0af88d"} Sep 10 09:14:48.479656 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:48.479517 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:48.479656 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:48.479575 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:48.479656 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:48.479623 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:48.494253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:48.494209 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:48.494356 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:48.494309 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:14:48.522835 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:48.522800 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" podStartSLOduration=8.349396596 podStartE2EDuration="26.522785785s" totalImagesPullingTime="18.173389189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:48.521612748 +0000 UTC m=+27.703301825" watchObservedRunningTime="2026-09-10 09:14:48.522785785 +0000 UTC m=+27.704474868" Sep 10 09:14:49.255426 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:49.255396 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:49.255426 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:49.255418 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:49.255592 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:49.255500 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:49.255592 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:49.255564 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:49.255655 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:49.255605 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:49.255716 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:49.255697 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:49.483534 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:49.483498 2546 generic.go:356] "Generic (PLEG): container finished" podID="bcdf75da-e1be-4a9d-a514-fbd4e04e4766" containerID="5bcab6371f1b95a1e0a82510db3bf76fe4af2590c02e0524e905f1215bccb59d" exitCode=0 Sep 10 09:14:49.484005 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:49.483581 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" event={"ID":"bcdf75da-e1be-4a9d-a514-fbd4e04e4766","Type":"ContainerDied","Data":"5bcab6371f1b95a1e0a82510db3bf76fe4af2590c02e0524e905f1215bccb59d"} Sep 10 09:14:50.181004 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.180246 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jr854"] Sep 10 09:14:50.181004 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.180431 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:50.181004 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:50.180547 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:50.183978 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.183602 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b49hb"] Sep 10 09:14:50.183978 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.183706 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:50.183978 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:50.183838 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:50.184201 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.184003 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-knj5j"] Sep 10 09:14:50.184201 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.184097 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:50.184201 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:50.184184 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:50.487633 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.487604 2546 generic.go:356] "Generic (PLEG): container finished" podID="bcdf75da-e1be-4a9d-a514-fbd4e04e4766" containerID="bcb7dd56ffb3b0d164451c2b527f3f74ca7e6d6ed2e5a0bad84c9f15b0d4d78e" exitCode=0 Sep 10 09:14:50.487913 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.487675 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" event={"ID":"bcdf75da-e1be-4a9d-a514-fbd4e04e4766","Type":"ContainerDied","Data":"bcb7dd56ffb3b0d164451c2b527f3f74ca7e6d6ed2e5a0bad84c9f15b0d4d78e"} Sep 10 09:14:50.500938 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.500915 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:50.569005 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:50.568980 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:51.170632 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:51.170597 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:51.256514 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:51.256491 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:51.256612 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:51.256569 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:51.490876 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:51.490847 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:51.857179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:51.857146 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:51.959434 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:51.959408 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:52.256070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:52.256046 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:52.256191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:52.256050 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:52.256238 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:52.256133 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:52.256304 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:52.256230 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:52.327149 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:52.327104 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:52.385214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:52.385188 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:52.494495 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:52.494469 2546 generic.go:356] "Generic (PLEG): container finished" podID="bcdf75da-e1be-4a9d-a514-fbd4e04e4766" containerID="ad185defc500852c4b7c776779a7ed9bb2aedcb7266c77bb20f0de9e217a336d" exitCode=0 Sep 10 09:14:52.494761 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:52.494528 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" event={"ID":"bcdf75da-e1be-4a9d-a514-fbd4e04e4766","Type":"ContainerDied","Data":"ad185defc500852c4b7c776779a7ed9bb2aedcb7266c77bb20f0de9e217a336d"} Sep 10 09:14:52.570073 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:52.569991 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:52.663338 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:52.663314 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:52.893835 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:52.893781 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:53.041352 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:53.041301 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:53.214671 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:53.214603 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mjg8s_0ed4f23e-1ad2-4346-8550-93565896543d/dns-node-resolver/0.log" Sep 10 09:14:53.255998 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:53.255968 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:53.256110 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:53.256087 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr854" podUID="ce6a92df-68a8-49fa-acb8-3b40e6852595" Sep 10 09:14:53.434212 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:53.434179 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:53.498729 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:53.498697 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:54.214346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:54.214166 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-htq4b_8615e2e1-f518-4237-960f-2eda0b516165/node-ca/0.log" Sep 10 09:14:54.247335 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:54.247300 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:54.255339 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:54.255315 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:54.255441 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:54.255408 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b49hb" podUID="52423870-4667-4de2-b4ee-6418b9edefaf" Sep 10 09:14:54.255496 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:54.255441 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:54.255553 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:54.255533 2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knj5j" podUID="ea243c6d-7906-4f8f-8da3-5e1d888d4d6a" Sep 10 09:14:54.313992 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:54.313966 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:54.933689 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:54.933662 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:54.934215 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:54.933815 2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:54.934215 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:54.933875 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret podName:ce6a92df-68a8-49fa-acb8-3b40e6852595 nodeName:}" failed. No retries permitted until 2026-09-10 09:15:26.933855293 +0000 UTC m=+66.115544352 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret") pod "global-pull-secret-syncer-jr854" (UID: "ce6a92df-68a8-49fa-acb8-3b40e6852595") : object "kube-system"/"original-pull-secret" not registered Sep 10 09:14:55.034931 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:55.034902 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:55.035243 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:55.035226 2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:55.035326 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:55.035302 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs podName:ea243c6d-7906-4f8f-8da3-5e1d888d4d6a nodeName:}" failed. No retries permitted until 2026-09-10 09:15:27.035270346 +0000 UTC m=+66.216959392 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs") pod "network-metrics-daemon-knj5j" (UID: "ea243c6d-7906-4f8f-8da3-5e1d888d4d6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 09:14:55.071608 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:55.071581 2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-170.ec2.internal" event="NodeReady" Sep 10 09:14:55.071720 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:55.071709 2546 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 09:14:55.093292 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:55.093242 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:55.136286 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:55.136257 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:55.136405 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:55.136383 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 09:14:55.136405 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:55.136399 2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 09:14:55.136492 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:55.136410 2546 projected.go:196] Error preparing data for projected volume kube-api-access-k28q6 for pod openshift-network-diagnostics/network-check-target-b49hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:55.136492 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:14:55.136467 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6 podName:52423870-4667-4de2-b4ee-6418b9edefaf nodeName:}" failed. No retries permitted until 2026-09-10 09:15:27.136449826 +0000 UTC m=+66.318138885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k28q6" (UniqueName: "kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6") pod "network-check-target-b49hb" (UID: "52423870-4667-4de2-b4ee-6418b9edefaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 09:14:55.255780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:55.255751 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:14:55.258178 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:55.258149 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 09:14:56.256117 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.256088 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:14:56.256588 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.256273 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:14:56.262020 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.261999 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 09:14:56.262155 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.262001 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 09:14:56.262663 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.262642 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k2pqr\"" Sep 10 09:14:56.262776 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.262684 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 09:14:56.262776 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.262691 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gfz2p\"" Sep 10 09:14:56.422474 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.422429 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:56.625428 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.625346 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:56.722512 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.722478 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:56.821403 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.821372 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:56.946542 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:56.946474 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:57.183771 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:57.183741 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:57.477922 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:57.477893 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:57.507454 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:57.507404 2546 generic.go:356] "Generic (PLEG): container finished" podID="bcdf75da-e1be-4a9d-a514-fbd4e04e4766" containerID="7bb5b1e91578d04263ab507a8289be9cc1bd8d77ab9493f527ccca5186eda588" exitCode=0 Sep 10 09:14:57.507539 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:57.507492 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" event={"ID":"bcdf75da-e1be-4a9d-a514-fbd4e04e4766","Type":"ContainerDied","Data":"7bb5b1e91578d04263ab507a8289be9cc1bd8d77ab9493f527ccca5186eda588"} Sep 10 09:14:57.913662 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:57.913582 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:58.512683 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:58.512654 2546 generic.go:356] "Generic (PLEG): container finished" podID="bcdf75da-e1be-4a9d-a514-fbd4e04e4766" containerID="daa71c9a7972d2a5d6ef79270815c0ab1e528b09e15d8a3f543798dcc4a99704" exitCode=0 Sep 10 09:14:58.513038 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:58.512705 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" event={"ID":"bcdf75da-e1be-4a9d-a514-fbd4e04e4766","Type":"ContainerDied","Data":"daa71c9a7972d2a5d6ef79270815c0ab1e528b09e15d8a3f543798dcc4a99704"} Sep 10 09:14:58.514918 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:58.514894 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:58.710092 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:58.710066 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:58.929012 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:58.928949 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:59.110117 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:59.110088 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:14:59.518659 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:59.518634 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" event={"ID":"bcdf75da-e1be-4a9d-a514-fbd4e04e4766","Type":"ContainerStarted","Data":"18c911bc9fbac98cc0caba9264b689622001ca6eb25165589bb2d26328d0958c"} Sep 10 09:14:59.543590 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:59.543550 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5tt2m" podStartSLOduration=7.841262822 podStartE2EDuration="37.543533618s" totalImagesPullingTime="29.702270796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:14:59.542822549 +0000 UTC m=+38.724511615" watchObservedRunningTime="2026-09-10 09:14:59.543533618 +0000 UTC m=+38.725222687" Sep 10 09:14:59.895585 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:14:59.895555 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:00.103849 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:00.103822 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:00.195377 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:00.195351 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:00.508378 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:00.508350 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:00.641712 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:00.641689 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:00.743178 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:00.743147 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:00.806255 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:00.806194 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:00.924070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:00.924042 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:01.137620 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:01.137540 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:01.308498 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:01.308470 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:01.926139 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:01.926110 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:02.269551 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:02.269521 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:02.369442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:02.369409 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:02.767014 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:02.766984 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:03.098753 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:03.098681 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:03.510334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:03.510302 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:04.410859 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:04.410819 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:04.511458 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:04.511424 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:04.706782 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:04.706717 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:05.099727 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:05.099694 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:05.505411 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:05.505375 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:05.903505 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:05.903437 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:06.316157 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:06.316128 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:06.710454 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:06.710395 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:06.901776 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:06.901744 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:07.111797 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:07.111772 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:08.428438 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:08.428396 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:09.220594 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:09.220557 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:15.542570 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:15.542520 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:15.639112 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:15.639054 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:15.983482 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:15.983443 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:16.039081 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:16.039051 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:16.194605 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:16.194573 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:16.296150 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:16.296052 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:16.730234 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:16.730203 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:16.832480 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:16.832442 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:17.287869 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.287830 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:17.456301 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.456247 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:17.490085 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.490049 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xd4qq"] Sep 10 09:15:17.491696 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.491678 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zfhl5"] Sep 10 09:15:17.491813 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.491795 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xd4qq" Sep 10 09:15:17.493462 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.493439 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.494184 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.494158 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 09:15:17.494316 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.494245 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jgrvm\"" Sep 10 09:15:17.495305 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.495266 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 09:15:17.495817 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.495799 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 09:15:17.496364 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.496342 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 09:15:17.496462 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.496382 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fcf7w\"" Sep 10 09:15:17.496462 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.496381 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 09:15:17.496551 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.496462 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 09:15:17.496551 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.496514 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 09:15:17.515632 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.515569 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xd4qq"] Sep 10 09:15:17.516334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.516271 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zfhl5"] Sep 10 09:15:17.520061 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.520036 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c2m2f"] Sep 10 09:15:17.521837 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.521819 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.525267 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.525238 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tpq75\"" Sep 10 09:15:17.525411 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.525236 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 09:15:17.525627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.525614 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 09:15:17.539738 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.539655 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79d77b6f48-zkjn8"] Sep 10 09:15:17.541889 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.541862 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.544449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.544422 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c2m2f"] Sep 10 09:15:17.545861 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.545783 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 09:15:17.546007 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.545800 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 09:15:17.546007 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.545960 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwmtf\"" Sep 10 09:15:17.546114 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.546027 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 09:15:17.551845 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.551815 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 09:15:17.571204 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.571164 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79d77b6f48-zkjn8"] Sep 10 09:15:17.585365 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585329 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5101d125-9003-4e52-8c8d-be6a11fabdbd-registry-tls\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.585365 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585368 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5101d125-9003-4e52-8c8d-be6a11fabdbd-installation-pull-secrets\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585391 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7961c4df-e598-459c-b080-e369916c1c64-metrics-tls\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585413 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzph\" (UniqueName: \"kubernetes.io/projected/34992b33-8b5a-4f69-9f8a-bc88b37f340a-kube-api-access-7xzph\") pod \"ingress-canary-xd4qq\" (UID: \"34992b33-8b5a-4f69-9f8a-bc88b37f340a\") " pod="openshift-ingress-canary/ingress-canary-xd4qq" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585435 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5101d125-9003-4e52-8c8d-be6a11fabdbd-ca-trust-extracted\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585497 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bad5d836-bb05-41eb-afc6-1fde320819f5-crio-socket\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585524 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5101d125-9003-4e52-8c8d-be6a11fabdbd-bound-sa-token\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585550 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34992b33-8b5a-4f69-9f8a-bc88b37f340a-cert\") pod \"ingress-canary-xd4qq\" (UID: \"34992b33-8b5a-4f69-9f8a-bc88b37f340a\") " pod="openshift-ingress-canary/ingress-canary-xd4qq" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585580 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bad5d836-bb05-41eb-afc6-1fde320819f5-data-volume\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585604 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7961c4df-e598-459c-b080-e369916c1c64-config-volume\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585621 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5101d125-9003-4e52-8c8d-be6a11fabdbd-registry-certificates\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585647 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtt5\" (UniqueName: \"kubernetes.io/projected/5101d125-9003-4e52-8c8d-be6a11fabdbd-kube-api-access-qxtt5\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585676 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7961c4df-e598-459c-b080-e369916c1c64-tmp-dir\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585690 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bad5d836-bb05-41eb-afc6-1fde320819f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585705 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5101d125-9003-4e52-8c8d-be6a11fabdbd-image-registry-private-configuration\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585727 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5101d125-9003-4e52-8c8d-be6a11fabdbd-trusted-ca\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.585805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585743 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwl2\" (UniqueName: \"kubernetes.io/projected/bad5d836-bb05-41eb-afc6-1fde320819f5-kube-api-access-dzwl2\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.586611 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585823 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bad5d836-bb05-41eb-afc6-1fde320819f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.586611 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.585841 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x69mp\" (UniqueName: \"kubernetes.io/projected/7961c4df-e598-459c-b080-e369916c1c64-kube-api-access-x69mp\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.602410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.602372 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:17.687025 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.686975 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtt5\" (UniqueName: \"kubernetes.io/projected/5101d125-9003-4e52-8c8d-be6a11fabdbd-kube-api-access-qxtt5\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.687025 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687032 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7961c4df-e598-459c-b080-e369916c1c64-tmp-dir\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.687341 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687052 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bad5d836-bb05-41eb-afc6-1fde320819f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.687341 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687170 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5101d125-9003-4e52-8c8d-be6a11fabdbd-image-registry-private-configuration\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.687341 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687206 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5101d125-9003-4e52-8c8d-be6a11fabdbd-trusted-ca\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.687341 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687228 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwl2\" (UniqueName: \"kubernetes.io/projected/bad5d836-bb05-41eb-afc6-1fde320819f5-kube-api-access-dzwl2\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.687542 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687352 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bad5d836-bb05-41eb-afc6-1fde320819f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.687542 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687390 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x69mp\" (UniqueName: \"kubernetes.io/projected/7961c4df-e598-459c-b080-e369916c1c64-kube-api-access-x69mp\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.687542 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687421 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5101d125-9003-4e52-8c8d-be6a11fabdbd-registry-tls\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.687542 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687450 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5101d125-9003-4e52-8c8d-be6a11fabdbd-installation-pull-secrets\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.687688 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687482 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7961c4df-e598-459c-b080-e369916c1c64-metrics-tls\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.688207 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687931 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzph\" (UniqueName: \"kubernetes.io/projected/34992b33-8b5a-4f69-9f8a-bc88b37f340a-kube-api-access-7xzph\") pod \"ingress-canary-xd4qq\" (UID: \"34992b33-8b5a-4f69-9f8a-bc88b37f340a\") " pod="openshift-ingress-canary/ingress-canary-xd4qq" Sep 10 09:15:17.688207 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687981 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5101d125-9003-4e52-8c8d-be6a11fabdbd-ca-trust-extracted\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.688207 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.688021 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bad5d836-bb05-41eb-afc6-1fde320819f5-crio-socket\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.688207 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.688052 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5101d125-9003-4e52-8c8d-be6a11fabdbd-bound-sa-token\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.688207 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.688083 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34992b33-8b5a-4f69-9f8a-bc88b37f340a-cert\") pod \"ingress-canary-xd4qq\" (UID: \"34992b33-8b5a-4f69-9f8a-bc88b37f340a\") " pod="openshift-ingress-canary/ingress-canary-xd4qq" Sep 10 09:15:17.688207 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.688113 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bad5d836-bb05-41eb-afc6-1fde320819f5-data-volume\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.688559 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.688131 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bad5d836-bb05-41eb-afc6-1fde320819f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.688559 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.688150 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7961c4df-e598-459c-b080-e369916c1c64-config-volume\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.689320 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.688841 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7961c4df-e598-459c-b080-e369916c1c64-config-volume\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.689320 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.688940 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bad5d836-bb05-41eb-afc6-1fde320819f5-crio-socket\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.689320 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.688973 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5101d125-9003-4e52-8c8d-be6a11fabdbd-registry-certificates\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.689320 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.689223 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5101d125-9003-4e52-8c8d-be6a11fabdbd-ca-trust-extracted\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.689892 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.689862 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bad5d836-bb05-41eb-afc6-1fde320819f5-data-volume\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.690410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.687487 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7961c4df-e598-459c-b080-e369916c1c64-tmp-dir\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.690930 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.690877 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5101d125-9003-4e52-8c8d-be6a11fabdbd-registry-certificates\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.694840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.694697 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5101d125-9003-4e52-8c8d-be6a11fabdbd-trusted-ca\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.694840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.694786 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bad5d836-bb05-41eb-afc6-1fde320819f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.694840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.694827 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5101d125-9003-4e52-8c8d-be6a11fabdbd-installation-pull-secrets\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.695082 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.694894 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34992b33-8b5a-4f69-9f8a-bc88b37f340a-cert\") pod \"ingress-canary-xd4qq\" (UID: \"34992b33-8b5a-4f69-9f8a-bc88b37f340a\") " pod="openshift-ingress-canary/ingress-canary-xd4qq" Sep 10 09:15:17.695082 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.694944 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7961c4df-e598-459c-b080-e369916c1c64-metrics-tls\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.695532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.695502 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5101d125-9003-4e52-8c8d-be6a11fabdbd-image-registry-private-configuration\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.695825 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.695807 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5101d125-9003-4e52-8c8d-be6a11fabdbd-registry-tls\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.700679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.700648 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtt5\" (UniqueName: \"kubernetes.io/projected/5101d125-9003-4e52-8c8d-be6a11fabdbd-kube-api-access-qxtt5\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.703221 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.703185 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzph\" (UniqueName: \"kubernetes.io/projected/34992b33-8b5a-4f69-9f8a-bc88b37f340a-kube-api-access-7xzph\") pod \"ingress-canary-xd4qq\" (UID: \"34992b33-8b5a-4f69-9f8a-bc88b37f340a\") " pod="openshift-ingress-canary/ingress-canary-xd4qq" Sep 10 09:15:17.703866 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.703839 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5101d125-9003-4e52-8c8d-be6a11fabdbd-bound-sa-token\") pod \"image-registry-79d77b6f48-zkjn8\" (UID: \"5101d125-9003-4e52-8c8d-be6a11fabdbd\") " pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.704755 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.704728 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x69mp\" (UniqueName: \"kubernetes.io/projected/7961c4df-e598-459c-b080-e369916c1c64-kube-api-access-x69mp\") pod \"dns-default-c2m2f\" (UID: \"7961c4df-e598-459c-b080-e369916c1c64\") " pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.706051 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.706032 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwl2\" (UniqueName: \"kubernetes.io/projected/bad5d836-bb05-41eb-afc6-1fde320819f5-kube-api-access-dzwl2\") pod \"insights-runtime-extractor-zfhl5\" (UID: \"bad5d836-bb05-41eb-afc6-1fde320819f5\") " pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.711140 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.711105 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:17.717356 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.717328 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v"] Sep 10 09:15:17.719908 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.719888 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" Sep 10 09:15:17.724106 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.724082 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4zpl6\"" Sep 10 09:15:17.724248 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.724080 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 09:15:17.730868 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.730838 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v"] Sep 10 09:15:17.790851 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.790760 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a7f1db2c-9409-44bd-8439-6810c5952810-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vq98v\" (UID: \"a7f1db2c-9409-44bd-8439-6810c5952810\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" Sep 10 09:15:17.802817 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.802764 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xd4qq" Sep 10 09:15:17.807514 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.807488 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zfhl5" Sep 10 09:15:17.831223 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.831191 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:17.852139 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.852051 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:17.891577 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.891535 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a7f1db2c-9409-44bd-8439-6810c5952810-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vq98v\" (UID: \"a7f1db2c-9409-44bd-8439-6810c5952810\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" Sep 10 09:15:17.897065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:17.896985 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a7f1db2c-9409-44bd-8439-6810c5952810-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vq98v\" (UID: \"a7f1db2c-9409-44bd-8439-6810c5952810\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" Sep 10 09:15:18.029864 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.029814 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" Sep 10 09:15:18.099481 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.099448 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xd4qq"] Sep 10 09:15:18.103187 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.102851 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zfhl5"] Sep 10 09:15:18.103519 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:18.103491 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34992b33_8b5a_4f69_9f8a_bc88b37f340a.slice/crio-54ea6b8b86c4c29b67bc2d173665cfc478631e695f6f8cb2ffb182372b7992fe WatchSource:0}: Error finding container 54ea6b8b86c4c29b67bc2d173665cfc478631e695f6f8cb2ffb182372b7992fe: Status 404 returned error can't find the container with id 54ea6b8b86c4c29b67bc2d173665cfc478631e695f6f8cb2ffb182372b7992fe Sep 10 09:15:18.107152 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:18.107128 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad5d836_bb05_41eb_afc6_1fde320819f5.slice/crio-54102535915e2d5cf53a38cf352e2324cabe349f6c636c4fc3ff5b45525ef5b9 WatchSource:0}: Error finding container 54102535915e2d5cf53a38cf352e2324cabe349f6c636c4fc3ff5b45525ef5b9: Status 404 returned error can't find the container with id 54102535915e2d5cf53a38cf352e2324cabe349f6c636c4fc3ff5b45525ef5b9 Sep 10 09:15:18.126739 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.126713 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c2m2f"] Sep 10 09:15:18.137451 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:18.137416 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7961c4df_e598_459c_b080_e369916c1c64.slice/crio-678f2635378993cd04b688e9c6bbd40633515defd7ebfe04a3fc0c127e4c966a WatchSource:0}: Error finding container 678f2635378993cd04b688e9c6bbd40633515defd7ebfe04a3fc0c127e4c966a: Status 404 returned error can't find the container with id 678f2635378993cd04b688e9c6bbd40633515defd7ebfe04a3fc0c127e4c966a Sep 10 09:15:18.146370 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.146338 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79d77b6f48-zkjn8"] Sep 10 09:15:18.153379 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:18.153348 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5101d125_9003_4e52_8c8d_be6a11fabdbd.slice/crio-00528b5e80ebdff39c49a2014d883f6e0d4c0e87c2b4ab1f68fa5b99197f3d34 WatchSource:0}: Error finding container 00528b5e80ebdff39c49a2014d883f6e0d4c0e87c2b4ab1f68fa5b99197f3d34: Status 404 returned error can't find the container with id 00528b5e80ebdff39c49a2014d883f6e0d4c0e87c2b4ab1f68fa5b99197f3d34 Sep 10 09:15:18.299159 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.299118 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v"] Sep 10 09:15:18.301970 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:18.301938 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f1db2c_9409_44bd_8439_6810c5952810.slice/crio-2b6ca3db8698498e9a37fafd9814d0c68ea606a7dd673428acc266f8484f01a5 WatchSource:0}: Error finding container 2b6ca3db8698498e9a37fafd9814d0c68ea606a7dd673428acc266f8484f01a5: Status 404 returned error can't find the container with id 2b6ca3db8698498e9a37fafd9814d0c68ea606a7dd673428acc266f8484f01a5 Sep 10 09:15:18.563525 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.563487 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xd4qq" event={"ID":"34992b33-8b5a-4f69-9f8a-bc88b37f340a","Type":"ContainerStarted","Data":"54ea6b8b86c4c29b67bc2d173665cfc478631e695f6f8cb2ffb182372b7992fe"} Sep 10 09:15:18.564587 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.564559 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2m2f" event={"ID":"7961c4df-e598-459c-b080-e369916c1c64","Type":"ContainerStarted","Data":"678f2635378993cd04b688e9c6bbd40633515defd7ebfe04a3fc0c127e4c966a"} Sep 10 09:15:18.565759 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.565731 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" event={"ID":"a7f1db2c-9409-44bd-8439-6810c5952810","Type":"ContainerStarted","Data":"2b6ca3db8698498e9a37fafd9814d0c68ea606a7dd673428acc266f8484f01a5"} Sep 10 09:15:18.567329 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.567273 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" event={"ID":"5101d125-9003-4e52-8c8d-be6a11fabdbd","Type":"ContainerStarted","Data":"0263c8e59de58cf32119746e7d18696889b69bd6698dc2faa99a15b6b5122d7f"} Sep 10 09:15:18.567422 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.567352 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" event={"ID":"5101d125-9003-4e52-8c8d-be6a11fabdbd","Type":"ContainerStarted","Data":"00528b5e80ebdff39c49a2014d883f6e0d4c0e87c2b4ab1f68fa5b99197f3d34"} Sep 10 09:15:18.567422 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.567383 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:18.568902 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.568883 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zfhl5" event={"ID":"bad5d836-bb05-41eb-afc6-1fde320819f5","Type":"ContainerStarted","Data":"e91d1cb12248e13a2636966b023a6ce92ab54140d4440cd3beb4222a820222bc"} Sep 10 09:15:18.568970 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.568921 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zfhl5" event={"ID":"bad5d836-bb05-41eb-afc6-1fde320819f5","Type":"ContainerStarted","Data":"54102535915e2d5cf53a38cf352e2324cabe349f6c636c4fc3ff5b45525ef5b9"} Sep 10 09:15:18.587858 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.587775 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" podStartSLOduration=1.5877489900000001 podStartE2EDuration="1.58774899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:18.587303547 +0000 UTC m=+57.768992630" watchObservedRunningTime="2026-09-10 09:15:18.58774899 +0000 UTC m=+57.769438096" Sep 10 09:15:18.846871 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.846604 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:18.947344 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:18.947304 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:20.384794 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:20.384705 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:20.446442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:20.446394 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:20.499702 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:20.499673 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xsql9" Sep 10 09:15:20.658451 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:20.658355 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:20.730739 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:20.730667 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:21.368090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.367994 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:21.474160 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.474129 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:21.580359 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.580321 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2m2f" event={"ID":"7961c4df-e598-459c-b080-e369916c1c64","Type":"ContainerStarted","Data":"6ec5049bb1e47a3f872a9f76096854e752a751f2053ec4a723030dface2418f3"} Sep 10 09:15:21.580555 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.580416 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2m2f" event={"ID":"7961c4df-e598-459c-b080-e369916c1c64","Type":"ContainerStarted","Data":"46da1a718ca7e244a8b12a54adc82a018c97a3bddede4a63c3cac7b935124216"} Sep 10 09:15:21.580555 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.580449 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:21.581929 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.581903 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" event={"ID":"a7f1db2c-9409-44bd-8439-6810c5952810","Type":"ContainerStarted","Data":"167fe8393a14e663cb7a8b86468c55400242a8859ab3306699f3717d23306160"} Sep 10 09:15:21.582038 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.582021 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" Sep 10 09:15:21.583977 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.583951 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zfhl5" event={"ID":"bad5d836-bb05-41eb-afc6-1fde320819f5","Type":"ContainerStarted","Data":"54a082a448db02a2f2faa62eac1c8ddda0944d7b7afe6a07661b5d45ac8f2184"} Sep 10 09:15:21.585465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.585441 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xd4qq" event={"ID":"34992b33-8b5a-4f69-9f8a-bc88b37f340a","Type":"ContainerStarted","Data":"ade7a1a45014058cdf9bc51c172bb775290d004b7f407520717b697dcf94fd11"} Sep 10 09:15:21.587151 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.587134 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" Sep 10 09:15:21.598988 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.598937 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c2m2f" podStartSLOduration=1.647271571 podStartE2EDuration="4.598921363s" totalImagesPullingTime="2.951649792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:21.597894839 +0000 UTC m=+60.779583932" watchObservedRunningTime="2026-09-10 09:15:21.598921363 +0000 UTC m=+60.780610423" Sep 10 09:15:21.615180 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.615121 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xd4qq" podStartSLOduration=1.6174828159999999 podStartE2EDuration="4.615103125s" totalImagesPullingTime="2.997620309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:21.614378303 +0000 UTC m=+60.796067389" watchObservedRunningTime="2026-09-10 09:15:21.615103125 +0000 UTC m=+60.796792191" Sep 10 09:15:21.641330 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.641204 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vq98v" podStartSLOduration=1.853967294 podStartE2EDuration="4.641186967s" totalImagesPullingTime="2.787219673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:21.640922005 +0000 UTC m=+60.822611067" watchObservedRunningTime="2026-09-10 09:15:21.641186967 +0000 UTC m=+60.822876032" Sep 10 09:15:21.723699 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.723660 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:21.820526 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.820466 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:21.914920 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:21.914818 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:23.423857 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:23.423806 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:23.481520 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:23.481481 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:23.595493 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:23.595454 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zfhl5" event={"ID":"bad5d836-bb05-41eb-afc6-1fde320819f5","Type":"ContainerStarted","Data":"7efd5d5cd33e110b6b0bb88ccee6cea10acbf0ddeb6880df42e53f879f7a8b47"} Sep 10 09:15:23.617028 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:23.616972 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zfhl5" podStartSLOduration=2.277981216 podStartE2EDuration="6.616953813s" totalImagesPullingTime="4.338972597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:15:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:23.616469184 +0000 UTC m=+62.798158250" watchObservedRunningTime="2026-09-10 09:15:23.616953813 +0000 UTC m=+62.798642878" Sep 10 09:15:24.479173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:24.479132 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:25.941840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:25.941793 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:26.143065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.143023 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:26.224736 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.224646 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-cmstd"] Sep 10 09:15:26.227723 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.227695 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.228354 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.228330 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-slhtd"] Sep 10 09:15:26.230192 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.230170 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.233017 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.232988 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 09:15:26.233155 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.233137 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 09:15:26.233525 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.233504 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 09:15:26.233525 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.233518 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 09:15:26.233712 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.233693 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 09:15:26.233774 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.233763 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 09:15:26.233997 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.233978 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 09:15:26.234146 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.234048 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 09:15:26.234299 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.234263 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-spccn\"" Sep 10 09:15:26.234393 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.234373 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-48b52\"" Sep 10 09:15:26.234444 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.234416 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 09:15:26.247081 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.247026 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-cmstd"] Sep 10 09:15:26.247586 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.247555 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:26.263072 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263037 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2754ca8d-cc92-4c02-9a54-89c57e442cba-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.263072 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263075 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c1e873e-f584-47d4-9f40-8282069fd755-root\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.263350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263097 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.263350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263116 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-textfile\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.263350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263135 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-wtmp\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.263350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263154 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.263350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263172 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwmw\" (UniqueName: \"kubernetes.io/projected/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-api-access-qgwmw\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.263350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263270 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmf2q\" (UniqueName: \"kubernetes.io/projected/8c1e873e-f584-47d4-9f40-8282069fd755-kube-api-access-kmf2q\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.263350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263331 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2754ca8d-cc92-4c02-9a54-89c57e442cba-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.263350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263351 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c1e873e-f584-47d4-9f40-8282069fd755-sys\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.263580 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263409 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.263580 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263458 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-tls\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.263580 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263484 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1e873e-f584-47d4-9f40-8282069fd755-metrics-client-ca\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.263580 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263500 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-accelerators-collector-config\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.263580 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.263521 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.346061 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.346024 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:26.364171 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364129 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2754ca8d-cc92-4c02-9a54-89c57e442cba-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.364350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364181 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c1e873e-f584-47d4-9f40-8282069fd755-root\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364241 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c1e873e-f584-47d4-9f40-8282069fd755-root\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364314 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.364501 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364354 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-textfile\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364501 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364372 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-wtmp\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364501 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364475 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-wtmp\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364501 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364490 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364702 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364519 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwmw\" (UniqueName: \"kubernetes.io/projected/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-api-access-qgwmw\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.364702 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364559 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmf2q\" (UniqueName: \"kubernetes.io/projected/8c1e873e-f584-47d4-9f40-8282069fd755-kube-api-access-kmf2q\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364702 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364592 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2754ca8d-cc92-4c02-9a54-89c57e442cba-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.364702 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364622 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c1e873e-f584-47d4-9f40-8282069fd755-sys\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364702 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364673 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.364945 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364702 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-textfile\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364945 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364719 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-tls\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364945 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364757 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1e873e-f584-47d4-9f40-8282069fd755-metrics-client-ca\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364945 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364771 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c1e873e-f584-47d4-9f40-8282069fd755-sys\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364945 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364782 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-accelerators-collector-config\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.364945 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.364821 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.365246 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.365019 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2754ca8d-cc92-4c02-9a54-89c57e442cba-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.365246 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.365093 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2754ca8d-cc92-4c02-9a54-89c57e442cba-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.365246 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:15:26.365122 2546 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 09:15:26.365246 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:15:26.365202 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-tls podName:8c1e873e-f584-47d4-9f40-8282069fd755 nodeName:}" failed. No retries permitted until 2026-09-10 09:15:26.865182362 +0000 UTC m=+66.046871410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-tls") pod "node-exporter-slhtd" (UID: "8c1e873e-f584-47d4-9f40-8282069fd755") : secret "node-exporter-tls" not found Sep 10 09:15:26.365478 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.365444 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.365478 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.365450 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c1e873e-f584-47d4-9f40-8282069fd755-metrics-client-ca\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.365742 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.365717 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-accelerators-collector-config\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.367095 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.367061 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.367206 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.367111 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.367381 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.367360 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.374215 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.374185 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmf2q\" (UniqueName: \"kubernetes.io/projected/8c1e873e-f584-47d4-9f40-8282069fd755-kube-api-access-kmf2q\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.374215 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.374197 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwmw\" (UniqueName: \"kubernetes.io/projected/2754ca8d-cc92-4c02-9a54-89c57e442cba-kube-api-access-qgwmw\") pod \"kube-state-metrics-74cb57df7-cmstd\" (UID: \"2754ca8d-cc92-4c02-9a54-89c57e442cba\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.518038 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.517999 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:26.538447 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.538404 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" Sep 10 09:15:26.615821 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.615785 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:26.777331 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.777225 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:26.810758 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.810718 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-cmstd"] Sep 10 09:15:26.813727 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:26.813693 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2754ca8d_cc92_4c02_9a54_89c57e442cba.slice/crio-0c3c63814620a969cb50912d8d2580e7b4db22e8f3d9f635f57d93625ca27335 WatchSource:0}: Error finding container 0c3c63814620a969cb50912d8d2580e7b4db22e8f3d9f635f57d93625ca27335: Status 404 returned error can't find the container with id 0c3c63814620a969cb50912d8d2580e7b4db22e8f3d9f635f57d93625ca27335 Sep 10 09:15:26.868240 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.868195 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-tls\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.870637 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.870610 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c1e873e-f584-47d4-9f40-8282069fd755-node-exporter-tls\") pod \"node-exporter-slhtd\" (UID: \"8c1e873e-f584-47d4-9f40-8282069fd755\") " pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:26.969124 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.969081 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:15:26.971458 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.971431 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 09:15:26.982487 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:26.982454 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce6a92df-68a8-49fa-acb8-3b40e6852595-original-pull-secret\") pod \"global-pull-secret-syncer-jr854\" (UID: \"ce6a92df-68a8-49fa-acb8-3b40e6852595\") " pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:15:27.019600 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.019560 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:27.066675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.066589 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr854" Sep 10 09:15:27.069503 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.069479 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:15:27.071680 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.071662 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 09:15:27.083073 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.083036 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea243c6d-7906-4f8f-8da3-5e1d888d4d6a-metrics-certs\") pod \"network-metrics-daemon-knj5j\" (UID: \"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a\") " pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:15:27.143765 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.143730 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slhtd" Sep 10 09:15:27.153576 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:27.153541 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1e873e_f584_47d4_9f40_8282069fd755.slice/crio-8121c318b27ca370e9a4f88e97e04264e1c711ca2b603339ee62edcbcd9c03da WatchSource:0}: Error finding container 8121c318b27ca370e9a4f88e97e04264e1c711ca2b603339ee62edcbcd9c03da: Status 404 returned error can't find the container with id 8121c318b27ca370e9a4f88e97e04264e1c711ca2b603339ee62edcbcd9c03da Sep 10 09:15:27.169483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.169454 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gfz2p\"" Sep 10 09:15:27.169805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.169781 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:15:27.171888 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.171772 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 09:15:27.178594 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.178548 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knj5j" Sep 10 09:15:27.182934 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.182908 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 09:15:27.194184 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.194157 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28q6\" (UniqueName: \"kubernetes.io/projected/52423870-4667-4de2-b4ee-6418b9edefaf-kube-api-access-k28q6\") pod \"network-check-target-b49hb\" (UID: \"52423870-4667-4de2-b4ee-6418b9edefaf\") " pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:15:27.235172 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.233479 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 09:15:27.242299 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.242260 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.244625 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.244597 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 09:15:27.244625 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.244612 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c49cl\"" Sep 10 09:15:27.245586 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.245427 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 09:15:27.245586 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.245438 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 09:15:27.245586 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.245445 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 09:15:27.245586 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.245486 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 09:15:27.245586 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.245437 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 09:15:27.246176 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.246157 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 09:15:27.246176 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.246167 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 09:15:27.246375 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.246190 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 09:15:27.252484 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.252075 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 09:15:27.338628 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.338552 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jr854"] Sep 10 09:15:27.341342 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:27.341311 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6a92df_68a8_49fa_acb8_3b40e6852595.slice/crio-5353c42965bc250cc16a256eb6d0eb6f9217f49dc434b30a6bfa4b1c9c548f4b WatchSource:0}: Error finding container 5353c42965bc250cc16a256eb6d0eb6f9217f49dc434b30a6bfa4b1c9c548f4b: Status 404 returned error can't find the container with id 5353c42965bc250cc16a256eb6d0eb6f9217f49dc434b30a6bfa4b1c9c548f4b Sep 10 09:15:27.371100 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371057 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-web-config\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371109 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klnw7\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-kube-api-access-klnw7\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371144 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371221 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-config-out\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371293 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371333 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371538 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371372 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371538 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371415 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371538 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371493 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371651 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371543 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371651 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371566 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371651 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371587 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.371651 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.371606 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.461516 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.461473 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-knj5j"] Sep 10 09:15:27.466744 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:27.466710 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea243c6d_7906_4f8f_8da3_5e1d888d4d6a.slice/crio-8880e26ccf251cec817e838a969dda76c702d36db8c0343a0f1dbb0ba749c0f7 WatchSource:0}: Error finding container 8880e26ccf251cec817e838a969dda76c702d36db8c0343a0f1dbb0ba749c0f7: Status 404 returned error can't find the container with id 8880e26ccf251cec817e838a969dda76c702d36db8c0343a0f1dbb0ba749c0f7 Sep 10 09:15:27.472683 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472654 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.472778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472708 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.472778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472736 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.472778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472763 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.472910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472779 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.472910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472805 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.472910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472829 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.472910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472885 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-web-config\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.473099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472922 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klnw7\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-kube-api-access-klnw7\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.473099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472960 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.473099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.472987 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-config-out\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.473099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.473043 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.473099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.473080 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.473341 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:15:27.473152 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-trusted-ca-bundle podName:c9d82569-ab0a-467a-b363-4a272d29162d nodeName:}" failed. No retries permitted until 2026-09-10 09:15:27.973125553 +0000 UTC m=+67.154814620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d") : configmap references non-existent config key: ca-bundle.crt Sep 10 09:15:27.473341 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:15:27.473266 2546 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 10 09:15:27.473448 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:15:27.473355 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-main-tls podName:c9d82569-ab0a-467a-b363-4a272d29162d nodeName:}" failed. No retries permitted until 2026-09-10 09:15:27.973336398 +0000 UTC m=+67.155025465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d") : secret "alertmanager-main-tls" not found Sep 10 09:15:27.473791 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.473691 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.473791 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.473764 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.475601 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.474903 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k2pqr\"" Sep 10 09:15:27.476032 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.476003 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.476124 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.476057 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.476350 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.476328 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.476483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.476439 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.476729 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.476708 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-config-out\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.476933 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.476915 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.477309 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.477258 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.478015 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.477993 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-web-config\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.481411 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.481381 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klnw7\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-kube-api-access-klnw7\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.482576 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.482547 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:15:27.610025 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.609924 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" event={"ID":"2754ca8d-cc92-4c02-9a54-89c57e442cba","Type":"ContainerStarted","Data":"0c3c63814620a969cb50912d8d2580e7b4db22e8f3d9f635f57d93625ca27335"} Sep 10 09:15:27.611111 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.611077 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-knj5j" event={"ID":"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a","Type":"ContainerStarted","Data":"8880e26ccf251cec817e838a969dda76c702d36db8c0343a0f1dbb0ba749c0f7"} Sep 10 09:15:27.612294 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.612253 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slhtd" event={"ID":"8c1e873e-f584-47d4-9f40-8282069fd755","Type":"ContainerStarted","Data":"8121c318b27ca370e9a4f88e97e04264e1c711ca2b603339ee62edcbcd9c03da"} Sep 10 09:15:27.613421 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.613399 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jr854" event={"ID":"ce6a92df-68a8-49fa-acb8-3b40e6852595","Type":"ContainerStarted","Data":"5353c42965bc250cc16a256eb6d0eb6f9217f49dc434b30a6bfa4b1c9c548f4b"} Sep 10 09:15:27.760256 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.760207 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b49hb"] Sep 10 09:15:27.763531 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:27.763489 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52423870_4667_4de2_b4ee_6418b9edefaf.slice/crio-a5ede09043709a0f4e6de33836a392c87bd9ea5efb9f6be1160174e59abbbfcb WatchSource:0}: Error finding container a5ede09043709a0f4e6de33836a392c87bd9ea5efb9f6be1160174e59abbbfcb: Status 404 returned error can't find the container with id a5ede09043709a0f4e6de33836a392c87bd9ea5efb9f6be1160174e59abbbfcb Sep 10 09:15:27.977805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.977707 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.977805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.977797 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.978718 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.978688 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:27.980928 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:27.980900 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:28.157853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:28.157566 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:15:28.218989 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:28.218948 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:28.336453 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:28.336241 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:28.466573 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:28.466452 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 09:15:28.619998 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:28.619951 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b49hb" event={"ID":"52423870-4667-4de2-b4ee-6418b9edefaf","Type":"ContainerStarted","Data":"a5ede09043709a0f4e6de33836a392c87bd9ea5efb9f6be1160174e59abbbfcb"} Sep 10 09:15:28.668887 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:28.668830 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d82569_ab0a_467a_b363_4a272d29162d.slice/crio-8d5c12c97a7c3795b8a24639288701b4ed14427fa0c48f8e3a474ed854c8724c WatchSource:0}: Error finding container 8d5c12c97a7c3795b8a24639288701b4ed14427fa0c48f8e3a474ed854c8724c: Status 404 returned error can't find the container with id 8d5c12c97a7c3795b8a24639288701b4ed14427fa0c48f8e3a474ed854c8724c Sep 10 09:15:29.233484 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.233170 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:29.315633 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.313818 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:29.426725 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.426633 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:29.489019 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.488866 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:29.632981 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.632762 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" event={"ID":"2754ca8d-cc92-4c02-9a54-89c57e442cba","Type":"ContainerStarted","Data":"82ed5928a5f757d9a18cfc9080159a6edd3ae79ec5045446bc54b6cdfb6f1fab"} Sep 10 09:15:29.632981 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.632880 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" event={"ID":"2754ca8d-cc92-4c02-9a54-89c57e442cba","Type":"ContainerStarted","Data":"d493afc7bc8585739adb8f41774544391ae5276d24e16f39f606a9d4b948031d"} Sep 10 09:15:29.632981 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.632909 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" event={"ID":"2754ca8d-cc92-4c02-9a54-89c57e442cba","Type":"ContainerStarted","Data":"8ce106c937402d09e863bce19ab66a9f7b8d8c2edc2f4c1540be28f47f8082fb"} Sep 10 09:15:29.637314 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.637252 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-knj5j" event={"ID":"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a","Type":"ContainerStarted","Data":"ff51c2b17c8a32bc76ba79f997144c411536d7e95f63000722eabe73d6d1c8a7"} Sep 10 09:15:29.637500 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.637345 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-knj5j" event={"ID":"ea243c6d-7906-4f8f-8da3-5e1d888d4d6a","Type":"ContainerStarted","Data":"1ce7fe61485f1bec9886f96cb149c4a6496cb5bb4e8114ff06a021216be7f198"} Sep 10 09:15:29.645439 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.645395 2546 generic.go:356] "Generic (PLEG): container finished" podID="8c1e873e-f584-47d4-9f40-8282069fd755" containerID="511023bbd4bdba9341cf66051a13e49274b7bcc4563a3fd018986d06a920ad2b" exitCode=0 Sep 10 09:15:29.645639 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.645526 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slhtd" event={"ID":"8c1e873e-f584-47d4-9f40-8282069fd755","Type":"ContainerDied","Data":"511023bbd4bdba9341cf66051a13e49274b7bcc4563a3fd018986d06a920ad2b"} Sep 10 09:15:29.649554 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.649459 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerStarted","Data":"8d5c12c97a7c3795b8a24639288701b4ed14427fa0c48f8e3a474ed854c8724c"} Sep 10 09:15:29.654038 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.653767 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-cmstd" podStartSLOduration=1.499039418 podStartE2EDuration="3.653743088s" totalImagesPullingTime="2.15470367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:29.652232423 +0000 UTC m=+68.833921516" watchObservedRunningTime="2026-09-10 09:15:29.653743088 +0000 UTC m=+68.835432155" Sep 10 09:15:29.784432 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.784335 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:29.883791 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:29.883748 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:30.656169 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.656083 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slhtd" event={"ID":"8c1e873e-f584-47d4-9f40-8282069fd755","Type":"ContainerStarted","Data":"dcc96889b5de3be3c9cea780902d06ff49e70145c7dd54ac9dbedaae625d0ac5"} Sep 10 09:15:30.656169 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.656155 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slhtd" event={"ID":"8c1e873e-f584-47d4-9f40-8282069fd755","Type":"ContainerStarted","Data":"6ab7b76b0bf5861177f7de09e5266ac01679700e88f258f04d48f1bd770ae9bc"} Sep 10 09:15:30.658148 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.658120 2546 generic.go:356] "Generic (PLEG): container finished" podID="c9d82569-ab0a-467a-b363-4a272d29162d" containerID="6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c" exitCode=0 Sep 10 09:15:30.658272 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.658212 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerDied","Data":"6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c"} Sep 10 09:15:30.686510 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.686440 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-knj5j" podStartSLOduration=67.183774688 podStartE2EDuration="1m8.686412228s" totalImagesPullingTime="1.50263754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:29.68549833 +0000 UTC m=+68.867187397" watchObservedRunningTime="2026-09-10 09:15:30.686412228 +0000 UTC m=+69.868101295" Sep 10 09:15:30.688777 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.688747 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-55d65cd66d-ckfrn"] Sep 10 09:15:30.697577 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.697544 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.700576 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.700543 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mbkw9\"" Sep 10 09:15:30.700576 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.700562 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 09:15:30.700783 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.700584 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 09:15:30.701250 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.701045 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 09:15:30.701250 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.701087 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 09:15:30.701250 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.701197 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b06efsvvi07rs\"" Sep 10 09:15:30.712316 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.712251 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-slhtd" podStartSLOduration=2.776879781 podStartE2EDuration="4.71223028s" totalImagesPullingTime="1.808760693s" totalInitContainerRuntime="126.589806ms" isStatefulPod=true podCreationTimestamp="2026-09-10 09:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:30.710370104 +0000 UTC m=+69.892059191" watchObservedRunningTime="2026-09-10 09:15:30.71223028 +0000 UTC m=+69.893919345" Sep 10 09:15:30.739915 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.739875 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55d65cd66d-ckfrn"] Sep 10 09:15:30.808130 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.808093 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2z4\" (UniqueName: \"kubernetes.io/projected/e97928d4-5581-4a1a-a354-a1a8e712bded-kube-api-access-4p2z4\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.808130 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.808142 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e97928d4-5581-4a1a-a354-a1a8e712bded-audit-log\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.808428 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.808224 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e97928d4-5581-4a1a-a354-a1a8e712bded-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.808428 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.808303 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e97928d4-5581-4a1a-a354-a1a8e712bded-secret-metrics-server-client-certs\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.808428 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.808399 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e97928d4-5581-4a1a-a354-a1a8e712bded-secret-metrics-server-tls\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.808562 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.808433 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e97928d4-5581-4a1a-a354-a1a8e712bded-client-ca-bundle\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.808562 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.808457 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e97928d4-5581-4a1a-a354-a1a8e712bded-metrics-server-audit-profiles\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.909727 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.909627 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e97928d4-5581-4a1a-a354-a1a8e712bded-secret-metrics-server-tls\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.909727 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.909680 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e97928d4-5581-4a1a-a354-a1a8e712bded-client-ca-bundle\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.909727 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.909709 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e97928d4-5581-4a1a-a354-a1a8e712bded-metrics-server-audit-profiles\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.910004 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.909736 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2z4\" (UniqueName: \"kubernetes.io/projected/e97928d4-5581-4a1a-a354-a1a8e712bded-kube-api-access-4p2z4\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.910004 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.909769 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e97928d4-5581-4a1a-a354-a1a8e712bded-audit-log\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.910004 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.909815 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e97928d4-5581-4a1a-a354-a1a8e712bded-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.910004 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.909845 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e97928d4-5581-4a1a-a354-a1a8e712bded-secret-metrics-server-client-certs\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.910857 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.910798 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e97928d4-5581-4a1a-a354-a1a8e712bded-audit-log\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.911083 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.911054 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e97928d4-5581-4a1a-a354-a1a8e712bded-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.911333 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.911269 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e97928d4-5581-4a1a-a354-a1a8e712bded-metrics-server-audit-profiles\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.913151 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.913126 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e97928d4-5581-4a1a-a354-a1a8e712bded-secret-metrics-server-tls\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.913253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.913160 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e97928d4-5581-4a1a-a354-a1a8e712bded-secret-metrics-server-client-certs\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.913798 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.913774 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e97928d4-5581-4a1a-a354-a1a8e712bded-client-ca-bundle\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:30.925661 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:30.925629 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2z4\" (UniqueName: \"kubernetes.io/projected/e97928d4-5581-4a1a-a354-a1a8e712bded-kube-api-access-4p2z4\") pod \"metrics-server-55d65cd66d-ckfrn\" (UID: \"e97928d4-5581-4a1a-a354-a1a8e712bded\") " pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:31.010893 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.010760 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:31.018105 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.018046 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:31.066122 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.066085 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266"] Sep 10 09:15:31.081952 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.081918 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:31.082422 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.082395 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266"] Sep 10 09:15:31.084299 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.084262 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hsdxk\"" Sep 10 09:15:31.084423 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.084367 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 09:15:31.131049 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.131004 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:31.212656 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.212566 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6dc4c77a-7daf-4279-b807-221620251b0a-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-d9266\" (UID: \"6dc4c77a-7daf-4279-b807-221620251b0a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:31.212656 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.212641 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgw72\" (UniqueName: \"kubernetes.io/projected/6dc4c77a-7daf-4279-b807-221620251b0a-kube-api-access-lgw72\") pod \"monitoring-plugin-85bbbbbf4-d9266\" (UID: \"6dc4c77a-7daf-4279-b807-221620251b0a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:31.310046 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.309991 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:31.314037 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.313995 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgw72\" (UniqueName: \"kubernetes.io/projected/6dc4c77a-7daf-4279-b807-221620251b0a-kube-api-access-lgw72\") pod \"monitoring-plugin-85bbbbbf4-d9266\" (UID: \"6dc4c77a-7daf-4279-b807-221620251b0a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:31.314241 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.314100 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6dc4c77a-7daf-4279-b807-221620251b0a-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-d9266\" (UID: \"6dc4c77a-7daf-4279-b807-221620251b0a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:31.316959 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.316925 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6dc4c77a-7daf-4279-b807-221620251b0a-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-d9266\" (UID: \"6dc4c77a-7daf-4279-b807-221620251b0a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:31.327212 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.327181 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgw72\" (UniqueName: \"kubernetes.io/projected/6dc4c77a-7daf-4279-b807-221620251b0a-kube-api-access-lgw72\") pod \"monitoring-plugin-85bbbbbf4-d9266\" (UID: \"6dc4c77a-7daf-4279-b807-221620251b0a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:31.394853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.394812 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:31.410638 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.410603 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:31.592960 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:31.592922 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c2m2f" Sep 10 09:15:32.297371 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:32.297213 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:32.417361 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:32.417317 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:33.678061 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:33.677959 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b49hb" event={"ID":"52423870-4667-4de2-b4ee-6418b9edefaf","Type":"ContainerStarted","Data":"86d76db33827f457fb60453d25ca569fef62eaaf2d8be3ae22bfb146a2cb6988"} Sep 10 09:15:33.678885 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:33.678108 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:15:33.681892 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:33.681828 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerStarted","Data":"ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b"} Sep 10 09:15:33.686703 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:33.684392 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jr854" event={"ID":"ce6a92df-68a8-49fa-acb8-3b40e6852595","Type":"ContainerStarted","Data":"ad1be0b13478e55ff9558d2d4f1f308a269c31c78abc0dc5a9da7236e084fc10"} Sep 10 09:15:33.693307 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:33.693206 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b49hb" podStartSLOduration=65.949990033 podStartE2EDuration="1m11.693171571s" totalImagesPullingTime="5.743181538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:33.692209979 +0000 UTC m=+72.873899086" watchObservedRunningTime="2026-09-10 09:15:33.693171571 +0000 UTC m=+72.874860637" Sep 10 09:15:33.710023 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:33.709901 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jr854" podStartSLOduration=65.543527099 podStartE2EDuration="1m11.709848197s" totalImagesPullingTime="6.166321098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:33.707879499 +0000 UTC m=+72.889568565" watchObservedRunningTime="2026-09-10 09:15:33.709848197 +0000 UTC m=+72.891537264" Sep 10 09:15:33.767303 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:33.767239 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266"] Sep 10 09:15:33.770249 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:33.770221 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55d65cd66d-ckfrn"] Sep 10 09:15:33.770718 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:33.770667 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc4c77a_7daf_4279_b807_221620251b0a.slice/crio-a811d54d738dd172d81d91a771b46a21a6c2bcc26117cf62727de1e064871550 WatchSource:0}: Error finding container a811d54d738dd172d81d91a771b46a21a6c2bcc26117cf62727de1e064871550: Status 404 returned error can't find the container with id a811d54d738dd172d81d91a771b46a21a6c2bcc26117cf62727de1e064871550 Sep 10 09:15:33.773231 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:33.773110 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97928d4_5581_4a1a_a354_a1a8e712bded.slice/crio-f6f5af262883ae48b2ec08519a6d4b42177a54dd845aad8dbeaa73c6b76c50be WatchSource:0}: Error finding container f6f5af262883ae48b2ec08519a6d4b42177a54dd845aad8dbeaa73c6b76c50be: Status 404 returned error can't find the container with id f6f5af262883ae48b2ec08519a6d4b42177a54dd845aad8dbeaa73c6b76c50be Sep 10 09:15:34.011162 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.011065 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69f694c8cd-gf754"] Sep 10 09:15:34.013168 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.013141 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.015493 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.015463 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 09:15:34.015653 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.015508 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 09:15:34.016087 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.016070 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 09:15:34.016602 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.016581 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 09:15:34.016602 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.016606 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 09:15:34.016919 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.016904 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kbcjh\"" Sep 10 09:15:34.017084 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.017058 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 09:15:34.017194 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.017181 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 09:15:34.022483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.022457 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 09:15:34.022627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.022542 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f694c8cd-gf754"] Sep 10 09:15:34.142570 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.142512 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-config\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.142570 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.142573 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-serving-cert\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.142793 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.142642 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-service-ca\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.142793 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.142706 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-oauth-serving-cert\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.142793 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.142725 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-trusted-ca-bundle\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.142883 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.142795 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-oauth-config\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.142883 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.142824 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv4mf\" (UniqueName: \"kubernetes.io/projected/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-kube-api-access-cv4mf\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.243571 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.243522 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-oauth-config\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.243571 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.243576 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4mf\" (UniqueName: \"kubernetes.io/projected/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-kube-api-access-cv4mf\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.243832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.243620 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-config\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.243832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.243645 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-serving-cert\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.243832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.243682 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-service-ca\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.243832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.243712 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-oauth-serving-cert\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.243832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.243739 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-trusted-ca-bundle\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.244550 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.244525 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-service-ca\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.244665 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.244583 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-config\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.244665 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.244583 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-oauth-serving-cert\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.244811 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.244791 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-trusted-ca-bundle\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.246169 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.246145 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-serving-cert\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.246261 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.246172 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-oauth-config\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.252127 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.252107 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4mf\" (UniqueName: \"kubernetes.io/projected/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-kube-api-access-cv4mf\") pod \"console-69f694c8cd-gf754\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.324576 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.324476 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:34.418832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.418796 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:34.520497 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.520458 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:34.594377 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.594257 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f694c8cd-gf754"] Sep 10 09:15:34.598934 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:15:34.598892 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e98c5c_2ba4_48cf_8d07_e4e49a3cbca0.slice/crio-f2259dc6b58c1f1429ba3fcc540dc04c7173ec5aa14c4501e44837eebdce6af5 WatchSource:0}: Error finding container f2259dc6b58c1f1429ba3fcc540dc04c7173ec5aa14c4501e44837eebdce6af5: Status 404 returned error can't find the container with id f2259dc6b58c1f1429ba3fcc540dc04c7173ec5aa14c4501e44837eebdce6af5 Sep 10 09:15:34.695562 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.695513 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerStarted","Data":"f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7"} Sep 10 09:15:34.696060 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.695614 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerStarted","Data":"60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5"} Sep 10 09:15:34.696060 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.695660 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerStarted","Data":"e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd"} Sep 10 09:15:34.696060 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.695704 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerStarted","Data":"836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed"} Sep 10 09:15:34.698039 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.697969 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f694c8cd-gf754" event={"ID":"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0","Type":"ContainerStarted","Data":"f2259dc6b58c1f1429ba3fcc540dc04c7173ec5aa14c4501e44837eebdce6af5"} Sep 10 09:15:34.699868 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.699731 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" event={"ID":"6dc4c77a-7daf-4279-b807-221620251b0a","Type":"ContainerStarted","Data":"a811d54d738dd172d81d91a771b46a21a6c2bcc26117cf62727de1e064871550"} Sep 10 09:15:34.701502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:34.701410 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" event={"ID":"e97928d4-5581-4a1a-a354-a1a8e712bded","Type":"ContainerStarted","Data":"f6f5af262883ae48b2ec08519a6d4b42177a54dd845aad8dbeaa73c6b76c50be"} Sep 10 09:15:35.711006 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:35.710958 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerStarted","Data":"bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0"} Sep 10 09:15:35.736927 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:35.736856 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.568559426 podStartE2EDuration="8.736832666s" totalImagesPullingTime="5.147550185s" totalInitContainerRuntime="20.723055ms" isStatefulPod=false podCreationTimestamp="2026-09-10 09:15:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:35.735611396 +0000 UTC m=+74.917300510" watchObservedRunningTime="2026-09-10 09:15:35.736832666 +0000 UTC m=+74.918521736" Sep 10 09:15:35.857318 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:35.857263 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:36.007852 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.007819 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:36.334441 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.334338 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:36.490852 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.490811 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:36.572250 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.572204 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:36.717011 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.716911 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" event={"ID":"6dc4c77a-7daf-4279-b807-221620251b0a","Type":"ContainerStarted","Data":"cdcdc523585f5ac67726d087c78b4c9ec54c0ca3b9a29b6056b79d2db3dfe9a8"} Sep 10 09:15:36.717503 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.717021 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:36.719341 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.719303 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" event={"ID":"e97928d4-5581-4a1a-a354-a1a8e712bded","Type":"ContainerStarted","Data":"2dc0c91b03106e3e9b034e6e89c4a470cfe171e21ddec44fa90c378e268f39b9"} Sep 10 09:15:36.722968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.722938 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" Sep 10 09:15:36.732119 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.732059 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-d9266" podStartSLOduration=3.704401803 podStartE2EDuration="5.732040786s" totalImagesPullingTime="2.027638983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:36.731682296 +0000 UTC m=+75.913371363" watchObservedRunningTime="2026-09-10 09:15:36.732040786 +0000 UTC m=+75.913729852" Sep 10 09:15:36.754888 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:36.754827 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" podStartSLOduration=4.7290293 podStartE2EDuration="6.754809255s" totalImagesPullingTime="2.025779955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:36.752834217 +0000 UTC m=+75.934523310" watchObservedRunningTime="2026-09-10 09:15:36.754809255 +0000 UTC m=+75.936498319" Sep 10 09:15:38.428371 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:38.428308 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:38.530405 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:38.530370 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:38.729345 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:38.729225 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f694c8cd-gf754" event={"ID":"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0","Type":"ContainerStarted","Data":"12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f"} Sep 10 09:15:38.746911 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:38.746853 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69f694c8cd-gf754" podStartSLOduration=2.042552839 podStartE2EDuration="5.746834379s" totalImagesPullingTime="3.70428154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:15:38.746309816 +0000 UTC m=+77.927998899" watchObservedRunningTime="2026-09-10 09:15:38.746834379 +0000 UTC m=+77.928523442" Sep 10 09:15:39.577225 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:39.577190 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79d77b6f48-zkjn8" Sep 10 09:15:39.700242 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:39.700205 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:39.800479 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:39.800439 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:39.901504 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:39.901413 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:40.297570 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:40.297518 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:40.361895 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:40.361861 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:41.888574 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:41.888535 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:41.988201 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:41.988165 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:43.212328 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:43.212293 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:43.313530 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:43.313497 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:44.324782 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:44.324648 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:44.324782 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:44.324718 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:44.329830 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:44.329802 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:44.758495 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:44.758463 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:15:44.989127 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:44.989087 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:45.089522 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:45.089424 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:45.188847 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:45.188801 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:45.288515 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:45.288473 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:45.390191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:45.390092 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:47.346211 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:47.346162 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:48.539301 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:48.539246 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:49.385779 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:49.385741 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:49.483003 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:49.482965 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:51.011208 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:51.011162 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:51.011643 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:51.011267 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:15:53.424954 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:53.424917 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:15:53.484956 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:15:53.484921 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:04.705298 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:04.705254 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b49hb" Sep 10 09:16:08.427166 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:08.427122 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:08.530082 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:08.530041 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:10.977797 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:10.977761 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:11.016939 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:11.016909 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:16:11.021158 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:11.021131 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-55d65cd66d-ckfrn" Sep 10 09:16:11.038626 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:11.038587 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:45.508645 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:45.508609 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:45.826890 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:45.826792 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:46.499158 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:46.499120 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 09:16:46.499453 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:46.499429 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="alertmanager" containerID="cri-o://ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b" gracePeriod=120 Sep 10 09:16:46.499539 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:46.499506 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy-metric" containerID="cri-o://f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7" gracePeriod=120 Sep 10 09:16:46.499598 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:46.499543 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="config-reloader" containerID="cri-o://836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed" gracePeriod=120 Sep 10 09:16:46.499657 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:46.499572 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy" containerID="cri-o://60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5" gracePeriod=120 Sep 10 09:16:46.499657 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:46.499537 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy-web" containerID="cri-o://e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd" gracePeriod=120 Sep 10 09:16:46.499657 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:46.499524 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="prom-label-proxy" containerID="cri-o://bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0" gracePeriod=120 Sep 10 09:16:47.000390 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000356 2546 generic.go:356] "Generic (PLEG): container finished" podID="c9d82569-ab0a-467a-b363-4a272d29162d" containerID="bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0" exitCode=0 Sep 10 09:16:47.000390 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000382 2546 generic.go:356] "Generic (PLEG): container finished" podID="c9d82569-ab0a-467a-b363-4a272d29162d" containerID="f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7" exitCode=0 Sep 10 09:16:47.000390 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000389 2546 generic.go:356] "Generic (PLEG): container finished" podID="c9d82569-ab0a-467a-b363-4a272d29162d" containerID="60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5" exitCode=0 Sep 10 09:16:47.000390 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000398 2546 generic.go:356] "Generic (PLEG): container finished" podID="c9d82569-ab0a-467a-b363-4a272d29162d" containerID="836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed" exitCode=0 Sep 10 09:16:47.000896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000406 2546 generic.go:356] "Generic (PLEG): container finished" podID="c9d82569-ab0a-467a-b363-4a272d29162d" containerID="ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b" exitCode=0 Sep 10 09:16:47.000896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000410 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerDied","Data":"bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0"} Sep 10 09:16:47.000896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000483 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerDied","Data":"f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7"} Sep 10 09:16:47.000896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000513 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerDied","Data":"60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5"} Sep 10 09:16:47.000896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000537 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerDied","Data":"836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed"} Sep 10 09:16:47.000896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.000560 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerDied","Data":"ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b"} Sep 10 09:16:47.737750 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.737716 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:47.846081 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.845980 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-tls-assets\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846081 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846036 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-main-db\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846081 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846063 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-config-out\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846081 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846082 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-main-tls\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846229 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-web-config\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846313 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klnw7\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-kube-api-access-klnw7\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846345 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-cluster-tls-config\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846371 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-metrics-client-ca\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846437 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-web\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846439 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:16:47.846762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846478 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-trusted-ca-bundle\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846539 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846571 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846597 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-config-volume\") pod \"c9d82569-ab0a-467a-b363-4a272d29162d\" (UID: \"c9d82569-ab0a-467a-b363-4a272d29162d\") " Sep 10 09:16:47.846954 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.846868 2546 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-main-db\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.847425 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.847100 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:16:47.847425 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.847311 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:16:47.849751 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.849718 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:16:47.850111 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.850065 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-config-out" (OuterVolumeSpecName: "config-out") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:16:47.850230 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.850113 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-config-volume" (OuterVolumeSpecName: "config-volume") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:16:47.850331 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.850265 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-kube-api-access-klnw7" (OuterVolumeSpecName: "kube-api-access-klnw7") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "kube-api-access-klnw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:16:47.850575 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.850539 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:16:47.850682 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.850651 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:16:47.850948 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.850927 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:16:47.851042 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.851019 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:16:47.854318 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.854231 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:16:47.861599 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.861560 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-web-config" (OuterVolumeSpecName: "web-config") pod "c9d82569-ab0a-467a-b363-4a272d29162d" (UID: "c9d82569-ab0a-467a-b363-4a272d29162d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:16:47.948110 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948057 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klnw7\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-kube-api-access-klnw7\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948110 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948097 2546 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-cluster-tls-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948110 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948109 2546 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-metrics-client-ca\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948110 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948121 2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948131 2546 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9d82569-ab0a-467a-b363-4a272d29162d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948141 2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948151 2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948160 2546 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-config-volume\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948168 2546 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9d82569-ab0a-467a-b363-4a272d29162d-tls-assets\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948175 2546 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9d82569-ab0a-467a-b363-4a272d29162d-config-out\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948185 2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-secret-alertmanager-main-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:47.948416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:47.948192 2546 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9d82569-ab0a-467a-b363-4a272d29162d-web-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:16:48.007672 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.007634 2546 generic.go:356] "Generic (PLEG): container finished" podID="c9d82569-ab0a-467a-b363-4a272d29162d" containerID="e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd" exitCode=0 Sep 10 09:16:48.008070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.007695 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerDied","Data":"e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd"} Sep 10 09:16:48.008070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.007718 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.008070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.007787 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9d82569-ab0a-467a-b363-4a272d29162d","Type":"ContainerDied","Data":"8d5c12c97a7c3795b8a24639288701b4ed14427fa0c48f8e3a474ed854c8724c"} Sep 10 09:16:48.008070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.007821 2546 scope.go:117] "RemoveContainer" containerID="bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0" Sep 10 09:16:48.016122 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.016094 2546 scope.go:117] "RemoveContainer" containerID="f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7" Sep 10 09:16:48.024195 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.024166 2546 scope.go:117] "RemoveContainer" containerID="60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5" Sep 10 09:16:48.032492 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.032466 2546 scope.go:117] "RemoveContainer" containerID="e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd" Sep 10 09:16:48.032637 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.032614 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 09:16:48.035779 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.035754 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 09:16:48.040449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.040422 2546 scope.go:117] "RemoveContainer" containerID="836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed" Sep 10 09:16:48.048486 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.048460 2546 scope.go:117] "RemoveContainer" containerID="ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b" Sep 10 09:16:48.056734 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.056712 2546 scope.go:117] "RemoveContainer" containerID="6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c" Sep 10 09:16:48.064780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.064747 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 09:16:48.065027 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065011 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy" Sep 10 09:16:48.065027 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065025 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065034 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="init-config-reloader" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065041 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="init-config-reloader" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065047 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy-web" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065053 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy-web" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065064 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="prom-label-proxy" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065069 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="prom-label-proxy" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065077 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="alertmanager" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065084 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="alertmanager" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065089 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy-metric" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065095 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy-metric" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065105 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="config-reloader" Sep 10 09:16:48.065173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065111 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="config-reloader" Sep 10 09:16:48.065751 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065185 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="alertmanager" Sep 10 09:16:48.065751 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065193 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="config-reloader" Sep 10 09:16:48.065751 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065200 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy" Sep 10 09:16:48.065751 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065207 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="prom-label-proxy" Sep 10 09:16:48.065751 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065216 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy-web" Sep 10 09:16:48.065751 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065222 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" containerName="kube-rbac-proxy-metric" Sep 10 09:16:48.065751 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065270 2546 scope.go:117] "RemoveContainer" containerID="bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0" Sep 10 09:16:48.065751 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:16:48.065697 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0\": container with ID starting with bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0 not found: ID does not exist" containerID="bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0" Sep 10 09:16:48.065751 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065725 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0"} err="failed to get container status \"bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0\": rpc error: code = NotFound desc = could not find container \"bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0\": container with ID starting with bb815fdf309309737142e57d21f77865dbce04b4d03f993a9e10236744847ac0 not found: ID does not exist" Sep 10 09:16:48.066097 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.065759 2546 scope.go:117] "RemoveContainer" containerID="f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7" Sep 10 09:16:48.066097 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:16:48.066063 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7\": container with ID starting with f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7 not found: ID does not exist" containerID="f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7" Sep 10 09:16:48.066320 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.066099 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7"} err="failed to get container status \"f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7\": rpc error: code = NotFound desc = could not find container \"f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7\": container with ID starting with f64ae7b4145118aa4ac534f819e4890f5543fc298917218a005c61f6193eb0e7 not found: ID does not exist" Sep 10 09:16:48.066320 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.066126 2546 scope.go:117] "RemoveContainer" containerID="60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5" Sep 10 09:16:48.066489 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:16:48.066468 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5\": container with ID starting with 60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5 not found: ID does not exist" containerID="60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5" Sep 10 09:16:48.066569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.066495 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5"} err="failed to get container status \"60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5\": rpc error: code = NotFound desc = could not find container \"60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5\": container with ID starting with 60cf63a97f3c54798fff1d18f32d80f48bbf50d2abfc540f5c901acf2a5ddfb5 not found: ID does not exist" Sep 10 09:16:48.066569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.066511 2546 scope.go:117] "RemoveContainer" containerID="e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd" Sep 10 09:16:48.066801 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:16:48.066782 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd\": container with ID starting with e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd not found: ID does not exist" containerID="e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd" Sep 10 09:16:48.066851 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.066806 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd"} err="failed to get container status \"e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd\": rpc error: code = NotFound desc = could not find container \"e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd\": container with ID starting with e867ce32af7d2be5a3c6c32ab3db754422aa24d6b006e9db32f77429a32a96dd not found: ID does not exist" Sep 10 09:16:48.066851 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.066822 2546 scope.go:117] "RemoveContainer" containerID="836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed" Sep 10 09:16:48.067083 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:16:48.067064 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed\": container with ID starting with 836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed not found: ID does not exist" containerID="836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed" Sep 10 09:16:48.067150 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.067091 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed"} err="failed to get container status \"836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed\": rpc error: code = NotFound desc = could not find container \"836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed\": container with ID starting with 836e72404963a4808b683e1dc21210faa9117b9f82a9efe57543f60b70494fed not found: ID does not exist" Sep 10 09:16:48.067150 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.067113 2546 scope.go:117] "RemoveContainer" containerID="ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b" Sep 10 09:16:48.067389 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:16:48.067372 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b\": container with ID starting with ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b not found: ID does not exist" containerID="ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b" Sep 10 09:16:48.067458 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.067393 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b"} err="failed to get container status \"ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b\": rpc error: code = NotFound desc = could not find container \"ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b\": container with ID starting with ecc74842f4fdaec66b21fce7895f3ce1b8eca0cfcbc2ace8476bcf120ed17f3b not found: ID does not exist" Sep 10 09:16:48.067458 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.067407 2546 scope.go:117] "RemoveContainer" containerID="6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c" Sep 10 09:16:48.067680 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:16:48.067662 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c\": container with ID starting with 6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c not found: ID does not exist" containerID="6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c" Sep 10 09:16:48.067731 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.067683 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c"} err="failed to get container status \"6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c\": rpc error: code = NotFound desc = could not find container \"6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c\": container with ID starting with 6408945d74972c5831b78e12aa34383107084fd508de368c936bbed281eadb2c not found: ID does not exist" Sep 10 09:16:48.068870 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.068856 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.071203 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.071179 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 09:16:48.071366 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.071179 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 09:16:48.071366 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.071187 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 09:16:48.071366 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.071270 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 09:16:48.071366 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.071227 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 09:16:48.071366 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.071227 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c49cl\"" Sep 10 09:16:48.071629 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.071571 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 09:16:48.071688 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.071667 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 09:16:48.071736 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.071711 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 09:16:48.076358 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.076332 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 09:16:48.087306 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.087239 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 09:16:48.149966 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.149863 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2c8fc12-a867-4cde-b675-9499a9d3abf7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.149966 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.149904 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2c8fc12-a867-4cde-b675-9499a9d3abf7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.149966 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.149959 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-web-config\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150003 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150042 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-config-volume\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150102 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150123 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150142 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150162 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150468 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150209 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c8fc12-a867-4cde-b675-9499a9d3abf7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150468 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150235 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e2c8fc12-a867-4cde-b675-9499a9d3abf7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150468 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150304 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2c8fc12-a867-4cde-b675-9499a9d3abf7-config-out\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.150468 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.150328 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgf2l\" (UniqueName: \"kubernetes.io/projected/e2c8fc12-a867-4cde-b675-9499a9d3abf7-kube-api-access-wgf2l\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.250913 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.250867 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-web-config\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.250913 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.250917 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251232 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.250938 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-config-volume\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251232 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.250974 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251232 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.250999 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251232 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.251025 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251232 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.251054 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251232 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.251196 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c8fc12-a867-4cde-b675-9499a9d3abf7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.251253 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e2c8fc12-a867-4cde-b675-9499a9d3abf7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.251329 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2c8fc12-a867-4cde-b675-9499a9d3abf7-config-out\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.251361 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgf2l\" (UniqueName: \"kubernetes.io/projected/e2c8fc12-a867-4cde-b675-9499a9d3abf7-kube-api-access-wgf2l\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.251390 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2c8fc12-a867-4cde-b675-9499a9d3abf7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.251610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.251414 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2c8fc12-a867-4cde-b675-9499a9d3abf7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.252194 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.252164 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2c8fc12-a867-4cde-b675-9499a9d3abf7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.253141 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.253111 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c8fc12-a867-4cde-b675-9499a9d3abf7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.253451 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.253427 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e2c8fc12-a867-4cde-b675-9499a9d3abf7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.254181 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.254151 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.254312 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.254197 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.254384 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.254319 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-web-config\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.254384 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.254329 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-config-volume\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.254710 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.254677 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.254814 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.254721 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.254909 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.254891 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e2c8fc12-a867-4cde-b675-9499a9d3abf7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.255461 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.255446 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e2c8fc12-a867-4cde-b675-9499a9d3abf7-config-out\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.256162 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.256146 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e2c8fc12-a867-4cde-b675-9499a9d3abf7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.262691 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.262665 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgf2l\" (UniqueName: \"kubernetes.io/projected/e2c8fc12-a867-4cde-b675-9499a9d3abf7-kube-api-access-wgf2l\") pod \"alertmanager-main-0\" (UID: \"e2c8fc12-a867-4cde-b675-9499a9d3abf7\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.380302 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.380238 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 09:16:48.660469 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:48.660430 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 09:16:48.663450 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:16:48.663421 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c8fc12_a867_4cde_b675_9499a9d3abf7.slice/crio-1aa2dcf3f2e96f7e42404e0db5197420b464971d5e21d03e99daf2f8df298319 WatchSource:0}: Error finding container 1aa2dcf3f2e96f7e42404e0db5197420b464971d5e21d03e99daf2f8df298319: Status 404 returned error can't find the container with id 1aa2dcf3f2e96f7e42404e0db5197420b464971d5e21d03e99daf2f8df298319 Sep 10 09:16:49.014806 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:49.014763 2546 generic.go:356] "Generic (PLEG): container finished" podID="e2c8fc12-a867-4cde-b675-9499a9d3abf7" containerID="016646f0a0844d987a42f9e023e544131b24a6813eff46dc5f51564450340ba1" exitCode=0 Sep 10 09:16:49.015348 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:49.014859 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2c8fc12-a867-4cde-b675-9499a9d3abf7","Type":"ContainerDied","Data":"016646f0a0844d987a42f9e023e544131b24a6813eff46dc5f51564450340ba1"} Sep 10 09:16:49.015348 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:49.014972 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2c8fc12-a867-4cde-b675-9499a9d3abf7","Type":"ContainerStarted","Data":"1aa2dcf3f2e96f7e42404e0db5197420b464971d5e21d03e99daf2f8df298319"} Sep 10 09:16:49.263837 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:49.263780 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9d82569-ab0a-467a-b363-4a272d29162d" path="/var/lib/kubelet/pods/c9d82569-ab0a-467a-b363-4a272d29162d/volumes" Sep 10 09:16:50.025707 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.025663 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2c8fc12-a867-4cde-b675-9499a9d3abf7","Type":"ContainerStarted","Data":"0a3b5aa7937a40e5b9a7f25eb126964851ac0f39c47269c8894099b9f68d4742"} Sep 10 09:16:50.026189 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.025754 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2c8fc12-a867-4cde-b675-9499a9d3abf7","Type":"ContainerStarted","Data":"ea34f0bd58030bba9a0149a7c05807d519dbd7c2a04fb6446bc3e9f252bbb455"} Sep 10 09:16:50.026189 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.025798 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2c8fc12-a867-4cde-b675-9499a9d3abf7","Type":"ContainerStarted","Data":"e93ca03e21242c64ed6d23816e50cd080a15784f3f476cae1e365b4c72050d52"} Sep 10 09:16:50.026189 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.025838 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2c8fc12-a867-4cde-b675-9499a9d3abf7","Type":"ContainerStarted","Data":"4aa33ec91692d6695cdda20d3e12db60e7f09867c6e498e84a9fbcaafa7c0167"} Sep 10 09:16:50.026189 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.025886 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2c8fc12-a867-4cde-b675-9499a9d3abf7","Type":"ContainerStarted","Data":"d34114dd2240d968e03d816cb21c87af2f78a86b74a83bd1ff55b553ef88aae3"} Sep 10 09:16:50.026189 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.025923 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e2c8fc12-a867-4cde-b675-9499a9d3abf7","Type":"ContainerStarted","Data":"bc8283d4b27d033609374ed848bb3fea44a7cca4ffe8ed43fed7838cd4d6e4df"} Sep 10 09:16:50.057797 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.057714 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.040996217 podStartE2EDuration="2.057687269s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.691052ms" isStatefulPod=false podCreationTimestamp="2026-09-10 09:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:16:50.055647109 +0000 UTC m=+149.237336174" watchObservedRunningTime="2026-09-10 09:16:50.057687269 +0000 UTC m=+149.239376337" Sep 10 09:16:50.589921 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.589881 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh"] Sep 10 09:16:50.592580 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.592559 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.594887 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.594852 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 09:16:50.595041 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.594923 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 09:16:50.595041 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.594947 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 09:16:50.595041 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.594962 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ln298\"" Sep 10 09:16:50.595041 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.594979 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 09:16:50.595257 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.595201 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 09:16:50.600962 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.600936 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 09:16:50.607215 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.607183 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh"] Sep 10 09:16:50.671488 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.671449 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlkpl\" (UniqueName: \"kubernetes.io/projected/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-kube-api-access-qlkpl\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.671676 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.671500 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-federate-client-tls\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.671676 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.671547 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-telemeter-client-tls\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.671676 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.671579 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-metrics-client-ca\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.671676 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.671648 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-secret-telemeter-client\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.671810 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.671705 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-serving-certs-ca-bundle\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.671810 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.671729 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.671810 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.671753 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.772500 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.772441 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-federate-client-tls\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.772500 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.772497 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-telemeter-client-tls\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.772734 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.772524 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-metrics-client-ca\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.772734 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.772544 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-secret-telemeter-client\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.772734 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.772585 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-serving-certs-ca-bundle\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.772734 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.772613 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.772917 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.772780 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.772917 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.772879 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlkpl\" (UniqueName: \"kubernetes.io/projected/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-kube-api-access-qlkpl\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.773508 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.773473 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-serving-certs-ca-bundle\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.773508 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.773492 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-metrics-client-ca\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.773695 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.773599 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.775201 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.775168 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-telemeter-client-tls\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.775349 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.775219 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-federate-client-tls\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.775673 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.775653 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-secret-telemeter-client\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.775711 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.775675 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.782434 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.782408 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlkpl\" (UniqueName: \"kubernetes.io/projected/1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c-kube-api-access-qlkpl\") pod \"telemeter-client-bcdcf84cb-9zrsh\" (UID: \"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c\") " pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:50.902775 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:50.902677 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" Sep 10 09:16:51.183049 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:51.183023 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh"] Sep 10 09:16:51.185008 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:16:51.184978 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1b6821_4f2b_44ab_9f51_bf7abe4b6c3c.slice/crio-00739743c3b32a1b86ad34547ce67f7c6e9377279387329aa3606a5e850ee74e WatchSource:0}: Error finding container 00739743c3b32a1b86ad34547ce67f7c6e9377279387329aa3606a5e850ee74e: Status 404 returned error can't find the container with id 00739743c3b32a1b86ad34547ce67f7c6e9377279387329aa3606a5e850ee74e Sep 10 09:16:52.035215 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:52.035169 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" event={"ID":"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c","Type":"ContainerStarted","Data":"00739743c3b32a1b86ad34547ce67f7c6e9377279387329aa3606a5e850ee74e"} Sep 10 09:16:53.611694 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:53.611663 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:53.711582 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:53.711538 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:54.046765 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.046726 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" event={"ID":"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c","Type":"ContainerStarted","Data":"edef71a43d64fdcf021405aef8223f8975422648db1ae0d745f4d32b1a337944"} Sep 10 09:16:54.046982 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.046787 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" event={"ID":"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c","Type":"ContainerStarted","Data":"bb8763bd87ecbf4399835af22eae967cca6eb4ce0e9a6a1e9e7c4a2edb2e4e14"} Sep 10 09:16:54.046982 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.046809 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" event={"ID":"1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c","Type":"ContainerStarted","Data":"d1d5d73453f4ba0dc65f3eb3ffd10ea0d6239d22c32e1492bc9ac293be33c0a6"} Sep 10 09:16:54.070294 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.070219 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-bcdcf84cb-9zrsh" podStartSLOduration=1.790842611 podStartE2EDuration="4.07020155s" totalImagesPullingTime="2.279358939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:16:54.068151017 +0000 UTC m=+153.249840126" watchObservedRunningTime="2026-09-10 09:16:54.07020155 +0000 UTC m=+153.251890616" Sep 10 09:16:54.164783 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.164731 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cc56d4cb8-j2k2k"] Sep 10 09:16:54.167079 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.167048 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.177784 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.177754 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cc56d4cb8-j2k2k"] Sep 10 09:16:54.304675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.304551 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-trusted-ca-bundle\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.304675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.304601 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-oauth-config\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.304675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.304628 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-console-config\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.304675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.304654 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-service-ca\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.304675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.304670 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjcv\" (UniqueName: \"kubernetes.io/projected/d13ea4c2-5ace-40dc-8708-61469f376489-kube-api-access-9bjcv\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.304978 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.304725 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-oauth-serving-cert\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.304978 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.304745 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-serving-cert\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.405752 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.405708 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-oauth-serving-cert\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.405752 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.405756 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-serving-cert\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.406053 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.405776 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-trusted-ca-bundle\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.406053 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.405797 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-oauth-config\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.406053 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.405921 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-console-config\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.406053 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.405960 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-service-ca\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.406053 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.405981 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjcv\" (UniqueName: \"kubernetes.io/projected/d13ea4c2-5ace-40dc-8708-61469f376489-kube-api-access-9bjcv\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.406673 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.406644 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-oauth-serving-cert\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.406788 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.406697 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-console-config\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.406788 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.406712 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-service-ca\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.406880 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.406859 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-trusted-ca-bundle\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.408507 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.408488 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-oauth-config\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.408586 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.408569 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-serving-cert\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.415046 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.415021 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjcv\" (UniqueName: \"kubernetes.io/projected/d13ea4c2-5ace-40dc-8708-61469f376489-kube-api-access-9bjcv\") pod \"console-6cc56d4cb8-j2k2k\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.478019 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.477965 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:16:54.752356 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:54.752317 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cc56d4cb8-j2k2k"] Sep 10 09:16:54.757421 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:16:54.757390 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13ea4c2_5ace_40dc_8708_61469f376489.slice/crio-e40322384afb7ba54df0c60dd6ca78f1baa2a08ef7fef7884d1b1009309888eb WatchSource:0}: Error finding container e40322384afb7ba54df0c60dd6ca78f1baa2a08ef7fef7884d1b1009309888eb: Status 404 returned error can't find the container with id e40322384afb7ba54df0c60dd6ca78f1baa2a08ef7fef7884d1b1009309888eb Sep 10 09:16:55.053111 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:55.053002 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc56d4cb8-j2k2k" event={"ID":"d13ea4c2-5ace-40dc-8708-61469f376489","Type":"ContainerStarted","Data":"1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0"} Sep 10 09:16:55.053111 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:55.053069 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc56d4cb8-j2k2k" event={"ID":"d13ea4c2-5ace-40dc-8708-61469f376489","Type":"ContainerStarted","Data":"e40322384afb7ba54df0c60dd6ca78f1baa2a08ef7fef7884d1b1009309888eb"} Sep 10 09:16:55.070589 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:55.070533 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cc56d4cb8-j2k2k" podStartSLOduration=1.070516316 podStartE2EDuration="1.070516316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:16:55.069710269 +0000 UTC m=+154.251399335" watchObservedRunningTime="2026-09-10 09:16:55.070516316 +0000 UTC m=+154.252205459" Sep 10 09:16:55.074438 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:55.074383 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:16:55.215388 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:16:55.215351 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:17:03.010304 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:03.010249 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:17:04.478704 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:04.478661 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:17:04.479098 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:04.478729 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:17:04.483693 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:04.483664 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:17:05.098736 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:05.098702 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:17:05.151183 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:05.151147 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69f694c8cd-gf754"] Sep 10 09:17:05.260896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:05.260858 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:17:05.360299 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:05.360191 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:17:05.460577 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:05.460533 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:17:30.172758 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.172695 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69f694c8cd-gf754" podUID="f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" containerName="console" containerID="cri-o://12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f" gracePeriod=15 Sep 10 09:17:30.420668 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.420636 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:17:30.514113 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514064 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-oauth-serving-cert\") pod \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " Sep 10 09:17:30.514346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514186 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-config\") pod \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " Sep 10 09:17:30.514346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514220 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-serving-cert\") pod \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " Sep 10 09:17:30.514346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514307 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-service-ca\") pod \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " Sep 10 09:17:30.514346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514326 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-oauth-config\") pod \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " Sep 10 09:17:30.514548 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514356 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-trusted-ca-bundle\") pod \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " Sep 10 09:17:30.514548 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514380 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv4mf\" (UniqueName: \"kubernetes.io/projected/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-kube-api-access-cv4mf\") pod \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\" (UID: \"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0\") " Sep 10 09:17:30.514711 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514622 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" (UID: "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:17:30.514789 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514746 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-config" (OuterVolumeSpecName: "console-config") pod "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" (UID: "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:17:30.514854 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514787 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" (UID: "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:17:30.514919 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.514860 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-service-ca" (OuterVolumeSpecName: "service-ca") pod "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" (UID: "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:17:30.516882 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.516848 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" (UID: "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:17:30.516882 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.516870 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" (UID: "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:17:30.517030 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.516909 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-kube-api-access-cv4mf" (OuterVolumeSpecName: "kube-api-access-cv4mf") pod "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" (UID: "f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0"). InnerVolumeSpecName "kube-api-access-cv4mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:17:30.615332 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.615260 2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-service-ca\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:17:30.615332 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.615326 2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-oauth-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:17:30.615332 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.615339 2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-trusted-ca-bundle\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:17:30.615574 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.615348 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv4mf\" (UniqueName: \"kubernetes.io/projected/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-kube-api-access-cv4mf\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:17:30.615574 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.615357 2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-oauth-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:17:30.615574 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.615365 2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:17:30.615574 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:30.615373 2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0-console-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:17:31.195730 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.195695 2546 generic.go:356] "Generic (PLEG): container finished" podID="f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" containerID="12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f" exitCode=0 Sep 10 09:17:31.196163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.195744 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f694c8cd-gf754" Sep 10 09:17:31.196163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.195786 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f694c8cd-gf754" event={"ID":"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0","Type":"ContainerDied","Data":"12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f"} Sep 10 09:17:31.196163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.195840 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f694c8cd-gf754" event={"ID":"f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0","Type":"ContainerDied","Data":"f2259dc6b58c1f1429ba3fcc540dc04c7173ec5aa14c4501e44837eebdce6af5"} Sep 10 09:17:31.196163 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.195862 2546 scope.go:117] "RemoveContainer" containerID="12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f" Sep 10 09:17:31.207230 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.207204 2546 scope.go:117] "RemoveContainer" containerID="12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f" Sep 10 09:17:31.207614 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:17:31.207588 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f\": container with ID starting with 12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f not found: ID does not exist" containerID="12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f" Sep 10 09:17:31.207697 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.207623 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f"} err="failed to get container status \"12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f\": rpc error: code = NotFound desc = could not find container \"12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f\": container with ID starting with 12ebc5b6e6940b48013e8381c294445cf7a8d773b2fd4a0d7b0f7f78dea84f9f not found: ID does not exist" Sep 10 09:17:31.217472 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.217427 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69f694c8cd-gf754"] Sep 10 09:17:31.224716 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.224667 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69f694c8cd-gf754"] Sep 10 09:17:31.260756 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:31.260717 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" path="/var/lib/kubelet/pods/f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0/volumes" Sep 10 09:17:36.651741 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:36.651700 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:17:53.335834 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:53.335779 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:17:53.436683 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:17:53.436636 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:00.710172 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:00.710132 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:00.844416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:00.844381 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:11.812065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.812026 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d6654b7bd-zgvhm"] Sep 10 09:18:11.812488 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.812402 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" containerName="console" Sep 10 09:18:11.812488 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.812416 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" containerName="console" Sep 10 09:18:11.812488 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.812473 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e98c5c-2ba4-48cf-8d07-e4e49a3cbca0" containerName="console" Sep 10 09:18:11.815549 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.815528 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.825682 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.825641 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d6654b7bd-zgvhm"] Sep 10 09:18:11.856230 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.856187 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-console-config\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.856230 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.856234 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-serving-cert\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.856531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.856255 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj5dq\" (UniqueName: \"kubernetes.io/projected/78346732-41e3-4278-9c47-6840f829136a-kube-api-access-jj5dq\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.856531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.856444 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-oauth-config\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.856531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.856497 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-service-ca\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.856531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.856521 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-oauth-serving-cert\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.856686 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.856584 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-trusted-ca-bundle\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.869246 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.869210 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:11.957402 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.957359 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-oauth-config\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.957402 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.957409 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-service-ca\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.957635 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.957428 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-oauth-serving-cert\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.957635 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.957461 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-trusted-ca-bundle\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.957635 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.957489 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-console-config\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.957635 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.957508 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-serving-cert\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.957906 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.957679 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5dq\" (UniqueName: \"kubernetes.io/projected/78346732-41e3-4278-9c47-6840f829136a-kube-api-access-jj5dq\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.958266 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.958238 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-service-ca\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.958389 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.958259 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-oauth-serving-cert\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.958389 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.958308 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-console-config\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.958602 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.958584 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-trusted-ca-bundle\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.960022 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.960002 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-oauth-config\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.960184 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.960164 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-serving-cert\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.966569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.966543 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5dq\" (UniqueName: \"kubernetes.io/projected/78346732-41e3-4278-9c47-6840f829136a-kube-api-access-jj5dq\") pod \"console-6d6654b7bd-zgvhm\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:11.970495 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:11.970462 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:12.125433 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:12.125328 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:12.400826 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:12.400736 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d6654b7bd-zgvhm"] Sep 10 09:18:12.405832 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:18:12.405794 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78346732_41e3_4278_9c47_6840f829136a.slice/crio-8ac36cbb0f66484ff4e2ddefb24d4d87130b5f70a39408a3fd2a0447e64f5a20 WatchSource:0}: Error finding container 8ac36cbb0f66484ff4e2ddefb24d4d87130b5f70a39408a3fd2a0447e64f5a20: Status 404 returned error can't find the container with id 8ac36cbb0f66484ff4e2ddefb24d4d87130b5f70a39408a3fd2a0447e64f5a20 Sep 10 09:18:12.711890 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:12.711798 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:12.777755 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:12.777719 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:13.352462 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:13.352420 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6654b7bd-zgvhm" event={"ID":"78346732-41e3-4278-9c47-6840f829136a","Type":"ContainerStarted","Data":"4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849"} Sep 10 09:18:13.352979 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:13.352475 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6654b7bd-zgvhm" event={"ID":"78346732-41e3-4278-9c47-6840f829136a","Type":"ContainerStarted","Data":"8ac36cbb0f66484ff4e2ddefb24d4d87130b5f70a39408a3fd2a0447e64f5a20"} Sep 10 09:18:13.371263 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:13.371206 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d6654b7bd-zgvhm" podStartSLOduration=2.371170112 podStartE2EDuration="2.371170112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:18:13.369603432 +0000 UTC m=+232.551292501" watchObservedRunningTime="2026-09-10 09:18:13.371170112 +0000 UTC m=+232.552859178" Sep 10 09:18:22.126469 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:22.126426 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:22.126871 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:22.126511 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:22.131415 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:22.131384 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:22.390304 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:22.390195 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:18:22.438229 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:22.438191 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cc56d4cb8-j2k2k"] Sep 10 09:18:22.678863 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:22.678779 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:22.736200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:22.736162 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:23.439968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:23.439932 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:23.541603 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:23.541566 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:18:47.461847 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.461728 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cc56d4cb8-j2k2k" podUID="d13ea4c2-5ace-40dc-8708-61469f376489" containerName="console" containerID="cri-o://1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0" gracePeriod=15 Sep 10 09:18:47.708709 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.708679 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:18:47.770378 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.770346 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bjcv\" (UniqueName: \"kubernetes.io/projected/d13ea4c2-5ace-40dc-8708-61469f376489-kube-api-access-9bjcv\") pod \"d13ea4c2-5ace-40dc-8708-61469f376489\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " Sep 10 09:18:47.770378 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.770395 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-oauth-config\") pod \"d13ea4c2-5ace-40dc-8708-61469f376489\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " Sep 10 09:18:47.770629 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.770416 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-oauth-serving-cert\") pod \"d13ea4c2-5ace-40dc-8708-61469f376489\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " Sep 10 09:18:47.770629 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.770469 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-trusted-ca-bundle\") pod \"d13ea4c2-5ace-40dc-8708-61469f376489\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " Sep 10 09:18:47.770629 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.770485 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-console-config\") pod \"d13ea4c2-5ace-40dc-8708-61469f376489\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " Sep 10 09:18:47.770629 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.770503 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-serving-cert\") pod \"d13ea4c2-5ace-40dc-8708-61469f376489\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " Sep 10 09:18:47.770629 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.770531 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-service-ca\") pod \"d13ea4c2-5ace-40dc-8708-61469f376489\" (UID: \"d13ea4c2-5ace-40dc-8708-61469f376489\") " Sep 10 09:18:47.771109 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.771039 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-service-ca" (OuterVolumeSpecName: "service-ca") pod "d13ea4c2-5ace-40dc-8708-61469f376489" (UID: "d13ea4c2-5ace-40dc-8708-61469f376489"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:18:47.771109 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.771050 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-console-config" (OuterVolumeSpecName: "console-config") pod "d13ea4c2-5ace-40dc-8708-61469f376489" (UID: "d13ea4c2-5ace-40dc-8708-61469f376489"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:18:47.771109 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.771054 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d13ea4c2-5ace-40dc-8708-61469f376489" (UID: "d13ea4c2-5ace-40dc-8708-61469f376489"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:18:47.771109 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.771094 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d13ea4c2-5ace-40dc-8708-61469f376489" (UID: "d13ea4c2-5ace-40dc-8708-61469f376489"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:18:47.772869 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.772833 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d13ea4c2-5ace-40dc-8708-61469f376489" (UID: "d13ea4c2-5ace-40dc-8708-61469f376489"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:18:47.773161 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.773141 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d13ea4c2-5ace-40dc-8708-61469f376489" (UID: "d13ea4c2-5ace-40dc-8708-61469f376489"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:18:47.773224 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.773164 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13ea4c2-5ace-40dc-8708-61469f376489-kube-api-access-9bjcv" (OuterVolumeSpecName: "kube-api-access-9bjcv") pod "d13ea4c2-5ace-40dc-8708-61469f376489" (UID: "d13ea4c2-5ace-40dc-8708-61469f376489"). InnerVolumeSpecName "kube-api-access-9bjcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:18:47.871497 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.871456 2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-oauth-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:18:47.871497 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.871490 2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-oauth-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:18:47.871497 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.871499 2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-trusted-ca-bundle\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:18:47.871497 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.871507 2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-console-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:18:47.871780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.871518 2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d13ea4c2-5ace-40dc-8708-61469f376489-console-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:18:47.871780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.871526 2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d13ea4c2-5ace-40dc-8708-61469f376489-service-ca\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:18:47.871780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:47.871535 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bjcv\" (UniqueName: \"kubernetes.io/projected/d13ea4c2-5ace-40dc-8708-61469f376489-kube-api-access-9bjcv\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:18:48.490531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:48.490489 2546 generic.go:356] "Generic (PLEG): container finished" podID="d13ea4c2-5ace-40dc-8708-61469f376489" containerID="1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0" exitCode=0 Sep 10 09:18:48.491000 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:48.490549 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc56d4cb8-j2k2k" Sep 10 09:18:48.491000 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:48.490564 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc56d4cb8-j2k2k" event={"ID":"d13ea4c2-5ace-40dc-8708-61469f376489","Type":"ContainerDied","Data":"1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0"} Sep 10 09:18:48.491000 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:48.490663 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc56d4cb8-j2k2k" event={"ID":"d13ea4c2-5ace-40dc-8708-61469f376489","Type":"ContainerDied","Data":"e40322384afb7ba54df0c60dd6ca78f1baa2a08ef7fef7884d1b1009309888eb"} Sep 10 09:18:48.491000 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:48.490694 2546 scope.go:117] "RemoveContainer" containerID="1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0" Sep 10 09:18:48.500154 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:48.500128 2546 scope.go:117] "RemoveContainer" containerID="1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0" Sep 10 09:18:48.500508 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:18:48.500485 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0\": container with ID starting with 1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0 not found: ID does not exist" containerID="1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0" Sep 10 09:18:48.500577 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:48.500520 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0"} err="failed to get container status \"1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0\": rpc error: code = NotFound desc = could not find container \"1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0\": container with ID starting with 1a92127b15c26a2036d37fe82ff1a338301934e96ba02d19b8e42a7412780df0 not found: ID does not exist" Sep 10 09:18:48.512382 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:48.512348 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cc56d4cb8-j2k2k"] Sep 10 09:18:48.516353 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:48.516321 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cc56d4cb8-j2k2k"] Sep 10 09:18:49.260611 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:18:49.260573 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d13ea4c2-5ace-40dc-8708-61469f376489" path="/var/lib/kubelet/pods/d13ea4c2-5ace-40dc-8708-61469f376489/volumes" Sep 10 09:19:05.202172 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:19:05.201887 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:19:21.190138 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:19:21.190109 2546 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 09:20:00.274232 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:00.274195 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:00.330326 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:00.330270 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:06.340311 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:06.340260 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:06.466529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:06.466489 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:31.221670 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.221586 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64dfb6bb58-9qbgh"] Sep 10 09:20:31.222144 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.221874 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13ea4c2-5ace-40dc-8708-61469f376489" containerName="console" Sep 10 09:20:31.222144 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.221885 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13ea4c2-5ace-40dc-8708-61469f376489" containerName="console" Sep 10 09:20:31.222144 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.221932 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d13ea4c2-5ace-40dc-8708-61469f376489" containerName="console" Sep 10 09:20:31.224843 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.224819 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.245118 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.245072 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64dfb6bb58-9qbgh"] Sep 10 09:20:31.282165 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.282124 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:31.315910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.315860 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4c646f4-ea2f-4622-aab5-97004536e447-console-oauth-config\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.315910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.315905 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-console-config\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.316133 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.315935 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-oauth-serving-cert\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.316133 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.315981 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-trusted-ca-bundle\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.316133 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.316022 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jxf\" (UniqueName: \"kubernetes.io/projected/f4c646f4-ea2f-4622-aab5-97004536e447-kube-api-access-48jxf\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.316133 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.316100 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-service-ca\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.316252 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.316139 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c646f4-ea2f-4622-aab5-97004536e447-console-serving-cert\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.339517 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.339478 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:31.416743 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.416696 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-trusted-ca-bundle\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.416904 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.416767 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jxf\" (UniqueName: \"kubernetes.io/projected/f4c646f4-ea2f-4622-aab5-97004536e447-kube-api-access-48jxf\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.416904 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.416821 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-service-ca\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.416904 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.416866 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c646f4-ea2f-4622-aab5-97004536e447-console-serving-cert\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.417057 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.416903 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4c646f4-ea2f-4622-aab5-97004536e447-console-oauth-config\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.417057 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.416928 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-console-config\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.417057 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.416953 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-oauth-serving-cert\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.417743 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.417711 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-oauth-serving-cert\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.417743 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.417738 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-service-ca\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.417916 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.417725 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-console-config\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.418035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.418018 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c646f4-ea2f-4622-aab5-97004536e447-trusted-ca-bundle\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.419677 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.419645 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4c646f4-ea2f-4622-aab5-97004536e447-console-oauth-config\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.419773 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.419652 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c646f4-ea2f-4622-aab5-97004536e447-console-serving-cert\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.426018 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.425983 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:31.427019 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.426997 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jxf\" (UniqueName: \"kubernetes.io/projected/f4c646f4-ea2f-4622-aab5-97004536e447-kube-api-access-48jxf\") pod \"console-64dfb6bb58-9qbgh\" (UID: \"f4c646f4-ea2f-4622-aab5-97004536e447\") " pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.487567 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.487530 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:31.534428 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.534389 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:31.810951 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.810893 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64dfb6bb58-9qbgh"] Sep 10 09:20:31.813685 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:20:31.813648 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c646f4_ea2f_4622_aab5_97004536e447.slice/crio-0a55612727ec6a3f999f781d57e69ad64ced47f4ab9c67bef17dba054a234d4e WatchSource:0}: Error finding container 0a55612727ec6a3f999f781d57e69ad64ced47f4ab9c67bef17dba054a234d4e: Status 404 returned error can't find the container with id 0a55612727ec6a3f999f781d57e69ad64ced47f4ab9c67bef17dba054a234d4e Sep 10 09:20:31.880423 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:31.880389 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64dfb6bb58-9qbgh" event={"ID":"f4c646f4-ea2f-4622-aab5-97004536e447","Type":"ContainerStarted","Data":"0a55612727ec6a3f999f781d57e69ad64ced47f4ab9c67bef17dba054a234d4e"} Sep 10 09:20:32.023902 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:32.023870 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:32.126710 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:32.126618 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:32.885399 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:32.885358 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64dfb6bb58-9qbgh" event={"ID":"f4c646f4-ea2f-4622-aab5-97004536e447","Type":"ContainerStarted","Data":"3f412ed98b51221535d6d92ceed54d76472783e1faea14295e29f654818c4b15"} Sep 10 09:20:32.902179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:32.902122 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64dfb6bb58-9qbgh" podStartSLOduration=1.9021016400000001 podStartE2EDuration="1.90210164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:20:32.901231497 +0000 UTC m=+372.082920606" watchObservedRunningTime="2026-09-10 09:20:32.90210164 +0000 UTC m=+372.083790705" Sep 10 09:20:37.408294 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:37.408244 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:37.508597 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:37.508560 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:38.154466 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.154416 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:38.167720 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.167681 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h"] Sep 10 09:20:38.171625 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.171587 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" Sep 10 09:20:38.173936 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.173899 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 10 09:20:38.174636 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.174598 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-xl8kc\"" Sep 10 09:20:38.175454 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.174620 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 10 09:20:38.176859 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.176833 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h"] Sep 10 09:20:38.278193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.278146 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758gz\" (UniqueName: \"kubernetes.io/projected/ca804916-6f4f-42c0-9c26-ed55673d1315-kube-api-access-758gz\") pod \"feast-operator-controller-manager-848686b66-z6h4h\" (UID: \"ca804916-6f4f-42c0-9c26-ed55673d1315\") " pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" Sep 10 09:20:38.379552 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.379507 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758gz\" (UniqueName: \"kubernetes.io/projected/ca804916-6f4f-42c0-9c26-ed55673d1315-kube-api-access-758gz\") pod \"feast-operator-controller-manager-848686b66-z6h4h\" (UID: \"ca804916-6f4f-42c0-9c26-ed55673d1315\") " pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" Sep 10 09:20:38.388198 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.388159 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-758gz\" (UniqueName: \"kubernetes.io/projected/ca804916-6f4f-42c0-9c26-ed55673d1315-kube-api-access-758gz\") pod \"feast-operator-controller-manager-848686b66-z6h4h\" (UID: \"ca804916-6f4f-42c0-9c26-ed55673d1315\") " pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" Sep 10 09:20:38.487944 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.487906 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" Sep 10 09:20:38.760158 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.760077 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h"] Sep 10 09:20:38.763554 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:20:38.763511 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca804916_6f4f_42c0_9c26_ed55673d1315.slice/crio-da214728c3f2b835205d20fc96048f5faea999fb23414dc1cd1e4338c568b6f6 WatchSource:0}: Error finding container da214728c3f2b835205d20fc96048f5faea999fb23414dc1cd1e4338c568b6f6: Status 404 returned error can't find the container with id da214728c3f2b835205d20fc96048f5faea999fb23414dc1cd1e4338c568b6f6 Sep 10 09:20:38.765821 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.765802 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 09:20:38.909613 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:38.909578 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" event={"ID":"ca804916-6f4f-42c0-9c26-ed55673d1315","Type":"ContainerStarted","Data":"da214728c3f2b835205d20fc96048f5faea999fb23414dc1cd1e4338c568b6f6"} Sep 10 09:20:41.534848 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:41.534804 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:41.535334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:41.534912 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:41.540234 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:41.540203 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:41.926944 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:41.926845 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64dfb6bb58-9qbgh" Sep 10 09:20:41.982731 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:41.982689 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d6654b7bd-zgvhm"] Sep 10 09:20:42.405717 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:42.405681 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:42.507506 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:42.507459 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:20:42.927943 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:42.927901 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" event={"ID":"ca804916-6f4f-42c0-9c26-ed55673d1315","Type":"ContainerStarted","Data":"c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c"} Sep 10 09:20:42.928407 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:42.927984 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" Sep 10 09:20:42.943832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:42.943773 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" podStartSLOduration=1.346590266 podStartE2EDuration="4.943754443s" totalImagesPullingTime="3.597164177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:20:42.943185855 +0000 UTC m=+382.124874973" watchObservedRunningTime="2026-09-10 09:20:42.943754443 +0000 UTC m=+382.125443508" Sep 10 09:20:53.934098 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:20:53.934068 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" Sep 10 09:21:07.009236 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.009182 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d6654b7bd-zgvhm" podUID="78346732-41e3-4278-9c47-6840f829136a" containerName="console" containerID="cri-o://4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849" gracePeriod=15 Sep 10 09:21:07.250099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.250071 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:21:07.328821 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.328718 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj5dq\" (UniqueName: \"kubernetes.io/projected/78346732-41e3-4278-9c47-6840f829136a-kube-api-access-jj5dq\") pod \"78346732-41e3-4278-9c47-6840f829136a\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " Sep 10 09:21:07.328821 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.328818 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-oauth-config\") pod \"78346732-41e3-4278-9c47-6840f829136a\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " Sep 10 09:21:07.329055 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.328856 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-console-config\") pod \"78346732-41e3-4278-9c47-6840f829136a\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " Sep 10 09:21:07.329055 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.328910 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-service-ca\") pod \"78346732-41e3-4278-9c47-6840f829136a\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " Sep 10 09:21:07.329055 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.328934 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-trusted-ca-bundle\") pod \"78346732-41e3-4278-9c47-6840f829136a\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " Sep 10 09:21:07.329055 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.328959 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-oauth-serving-cert\") pod \"78346732-41e3-4278-9c47-6840f829136a\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " Sep 10 09:21:07.329055 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.328990 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-serving-cert\") pod \"78346732-41e3-4278-9c47-6840f829136a\" (UID: \"78346732-41e3-4278-9c47-6840f829136a\") " Sep 10 09:21:07.329415 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.329382 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-service-ca" (OuterVolumeSpecName: "service-ca") pod "78346732-41e3-4278-9c47-6840f829136a" (UID: "78346732-41e3-4278-9c47-6840f829136a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:21:07.329513 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.329490 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "78346732-41e3-4278-9c47-6840f829136a" (UID: "78346732-41e3-4278-9c47-6840f829136a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:21:07.329579 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.329402 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "78346732-41e3-4278-9c47-6840f829136a" (UID: "78346732-41e3-4278-9c47-6840f829136a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:21:07.329619 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.329578 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-console-config" (OuterVolumeSpecName: "console-config") pod "78346732-41e3-4278-9c47-6840f829136a" (UID: "78346732-41e3-4278-9c47-6840f829136a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:21:07.331228 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.331203 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "78346732-41e3-4278-9c47-6840f829136a" (UID: "78346732-41e3-4278-9c47-6840f829136a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:21:07.331546 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.331521 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "78346732-41e3-4278-9c47-6840f829136a" (UID: "78346732-41e3-4278-9c47-6840f829136a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:21:07.331546 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.331532 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78346732-41e3-4278-9c47-6840f829136a-kube-api-access-jj5dq" (OuterVolumeSpecName: "kube-api-access-jj5dq") pod "78346732-41e3-4278-9c47-6840f829136a" (UID: "78346732-41e3-4278-9c47-6840f829136a"). InnerVolumeSpecName "kube-api-access-jj5dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:21:07.429806 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.429761 2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-console-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:21:07.429806 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.429797 2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-service-ca\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:21:07.429806 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.429809 2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-trusted-ca-bundle\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:21:07.430065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.429822 2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78346732-41e3-4278-9c47-6840f829136a-oauth-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:21:07.430065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.429833 2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-serving-cert\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:21:07.430065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.429846 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj5dq\" (UniqueName: \"kubernetes.io/projected/78346732-41e3-4278-9c47-6840f829136a-kube-api-access-jj5dq\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:21:07.430065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.429858 2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78346732-41e3-4278-9c47-6840f829136a-console-oauth-config\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:21:07.716306 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.716202 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:21:07.775973 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:07.775933 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:21:08.026334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:08.026298 2546 generic.go:356] "Generic (PLEG): container finished" podID="78346732-41e3-4278-9c47-6840f829136a" containerID="4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849" exitCode=0 Sep 10 09:21:08.026334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:08.026338 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6654b7bd-zgvhm" event={"ID":"78346732-41e3-4278-9c47-6840f829136a","Type":"ContainerDied","Data":"4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849"} Sep 10 09:21:08.026799 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:08.026355 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6654b7bd-zgvhm" Sep 10 09:21:08.026799 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:08.026378 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6654b7bd-zgvhm" event={"ID":"78346732-41e3-4278-9c47-6840f829136a","Type":"ContainerDied","Data":"8ac36cbb0f66484ff4e2ddefb24d4d87130b5f70a39408a3fd2a0447e64f5a20"} Sep 10 09:21:08.026799 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:08.026399 2546 scope.go:117] "RemoveContainer" containerID="4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849" Sep 10 09:21:08.035899 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:08.035879 2546 scope.go:117] "RemoveContainer" containerID="4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849" Sep 10 09:21:08.036213 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:21:08.036195 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849\": container with ID starting with 4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849 not found: ID does not exist" containerID="4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849" Sep 10 09:21:08.036271 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:08.036223 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849"} err="failed to get container status \"4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849\": rpc error: code = NotFound desc = could not find container \"4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849\": container with ID starting with 4029c4e5ac1d258c889363253cfb233f4a59fcd36d09e25b087d6dd661aa0849 not found: ID does not exist" Sep 10 09:21:08.046836 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:08.046803 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d6654b7bd-zgvhm"] Sep 10 09:21:08.050539 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:08.050511 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d6654b7bd-zgvhm"] Sep 10 09:21:09.260559 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:09.260516 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78346732-41e3-4278-9c47-6840f829136a" path="/var/lib/kubelet/pods/78346732-41e3-4278-9c47-6840f829136a/volumes" Sep 10 09:21:14.975311 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:14.975231 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:21:15.728653 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.728615 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx"] Sep 10 09:21:15.728922 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.728909 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78346732-41e3-4278-9c47-6840f829136a" containerName="console" Sep 10 09:21:15.728922 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.728922 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="78346732-41e3-4278-9c47-6840f829136a" containerName="console" Sep 10 09:21:15.729146 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.728986 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="78346732-41e3-4278-9c47-6840f829136a" containerName="console" Sep 10 09:21:15.731262 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.731245 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.733587 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.733561 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 10 09:21:15.733766 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.733744 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 10 09:21:15.733848 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.733818 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 10 09:21:15.733913 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.733842 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-6m9ks\"" Sep 10 09:21:15.733913 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.733848 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 10 09:21:15.733913 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.733859 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 10 09:21:15.734192 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.734178 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 10 09:21:15.734238 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.734210 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 10 09:21:15.748397 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.748364 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx"] Sep 10 09:21:15.800010 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.799963 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-offline-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.800010 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.800006 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-registry-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.800312 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.800106 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-online-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.800312 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.800158 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxnqn\" (UniqueName: \"kubernetes.io/projected/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-kube-api-access-rxnqn\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.800312 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.800188 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-ui-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.800312 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.800203 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-feast-data\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.863819 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.863768 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:21:15.900917 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.900871 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-online-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.901089 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.900930 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxnqn\" (UniqueName: \"kubernetes.io/projected/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-kube-api-access-rxnqn\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.901089 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.900976 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-ui-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.901089 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.900998 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-feast-data\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.901089 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.901026 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-offline-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.901089 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.901053 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-registry-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.901362 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:21:15.901151 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 10 09:21:15.901362 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:21:15.901227 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-offline-tls podName:e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8 nodeName:}" failed. No retries permitted until 2026-09-10 09:21:16.401207836 +0000 UTC m=+415.582896898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-offline-tls") pod "feast-simple-feast-setup-6fc95c464f-wj2xx" (UID: "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8") : secret "feast-simple-feast-setup-offline-tls" not found Sep 10 09:21:15.901362 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:21:15.901255 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 10 09:21:15.901489 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:21:15.901367 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-online-tls podName:e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8 nodeName:}" failed. No retries permitted until 2026-09-10 09:21:16.401347618 +0000 UTC m=+415.583036680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-online-tls") pod "feast-simple-feast-setup-6fc95c464f-wj2xx" (UID: "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8") : secret "feast-simple-feast-setup-online-tls" not found Sep 10 09:21:15.901489 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.901455 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-feast-data\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.903756 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.903728 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-ui-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.903756 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.903739 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-registry-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.911774 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.911744 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxnqn\" (UniqueName: \"kubernetes.io/projected/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-kube-api-access-rxnqn\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:15.967448 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:15.967409 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:21:16.085856 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:16.085813 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:21:16.187361 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:16.187315 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:21:16.405428 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:16.405335 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-offline-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:16.405428 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:16.405409 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-online-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:16.407965 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:16.407937 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-online-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:16.408114 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:16.408003 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-offline-tls\") pod \"feast-simple-feast-setup-6fc95c464f-wj2xx\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:16.651718 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:16.651676 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:16.935134 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:16.935050 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx"] Sep 10 09:21:16.938899 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:21:16.938867 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode16c486a_ff9f_4ffd_aceb_d6a1fe532fd8.slice/crio-08661ee23ebc8292b77db1c2336e7e8d0b771e227afcdc72c11512eeb707212d WatchSource:0}: Error finding container 08661ee23ebc8292b77db1c2336e7e8d0b771e227afcdc72c11512eeb707212d: Status 404 returned error can't find the container with id 08661ee23ebc8292b77db1c2336e7e8d0b771e227afcdc72c11512eeb707212d Sep 10 09:21:17.065613 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:17.065573 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerStarted","Data":"08661ee23ebc8292b77db1c2336e7e8d0b771e227afcdc72c11512eeb707212d"} Sep 10 09:21:44.181456 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:44.181408 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerStarted","Data":"c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557"} Sep 10 09:21:49.201679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:49.201645 2546 generic.go:356] "Generic (PLEG): container finished" podID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerID="c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557" exitCode=0 Sep 10 09:21:49.202058 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:49.201719 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerDied","Data":"c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557"} Sep 10 09:21:50.207441 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:50.207403 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerStarted","Data":"74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac"} Sep 10 09:21:55.229352 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:55.229192 2546 generic.go:356] "Generic (PLEG): container finished" podID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerID="74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac" exitCode=0 Sep 10 09:21:55.229352 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:55.229269 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerDied","Data":"74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac"} Sep 10 09:21:55.238501 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:55.238481 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 10 09:21:56.239187 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:56.239148 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerStarted","Data":"b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3"} Sep 10 09:21:56.239716 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:56.239211 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerStarted","Data":"931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506"} Sep 10 09:21:56.239716 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:56.239233 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerStarted","Data":"3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d"} Sep 10 09:21:56.239716 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:56.239252 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerStarted","Data":"613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b"} Sep 10 09:21:56.270014 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:56.268504 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podStartSLOduration=14.614703347 podStartE2EDuration="41.268480322s" totalImagesPullingTime="26.653776975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:21:56.267472175 +0000 UTC m=+455.449161244" watchObservedRunningTime="2026-09-10 09:21:56.268480322 +0000 UTC m=+455.450169390" Sep 10 09:21:56.652181 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:56.652082 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:56.652472 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:56.652434 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:56.652472 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:56.652476 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:56.652669 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:56.652497 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:58.652325 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:58.652177 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:58.652325 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:58.652246 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:58.652325 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:58.652270 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:58.652926 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:58.652416 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:21:58.654674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:58.654602 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.22:8816: connect: connection refused" Sep 10 09:21:58.654846 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:58.654677 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 10 09:21:58.654846 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:58.654635 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 10 09:21:58.654846 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:21:58.654629 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.22:6571: connect: connection refused" Sep 10 09:22:01.652516 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:01.652460 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.22:8816: connect: connection refused" Sep 10 09:22:01.653041 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:01.652472 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.22:6571: connect: connection refused" Sep 10 09:22:01.653041 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:01.652472 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 10 09:22:01.653041 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:01.652472 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 10 09:22:04.653627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:04.653594 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:22:04.654152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:04.653777 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:22:04.654152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:04.653904 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:22:04.654443 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:04.654416 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:22:04.654551 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:04.654495 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:22:04.654833 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:04.654814 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:22:04.658788 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:04.658765 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:22:05.291253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:05.291220 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:22:05.674744 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:22:05.674626 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:23:07.939113 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:07.939072 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:23:07.949942 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:07.949905 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx"] Sep 10 09:23:07.950197 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:07.950165 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="registry" containerID="cri-o://613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b" gracePeriod=30 Sep 10 09:23:07.950459 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:07.950233 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="online" containerID="cri-o://3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d" gracePeriod=30 Sep 10 09:23:07.950573 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:07.950220 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="offline" containerID="cri-o://931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506" gracePeriod=30 Sep 10 09:23:07.950659 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:07.950238 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="ui" containerID="cri-o://b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3" gracePeriod=30 Sep 10 09:23:07.995875 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:07.995839 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:23:08.238417 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.238327 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:23:08.337906 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.337864 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:23:08.364199 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.364154 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g"] Sep 10 09:23:08.367379 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.367351 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.370015 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.369990 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-49ddt\"" Sep 10 09:23:08.381088 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.381049 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g"] Sep 10 09:23:08.402786 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.402729 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dckbx\" (UniqueName: \"kubernetes.io/projected/cb80455e-f8e1-4e88-89e1-b6f057c5413c-kube-api-access-dckbx\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.403013 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.402852 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.403013 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.402879 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80455e-f8e1-4e88-89e1-b6f057c5413c-feast-data\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.403013 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.402923 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-online-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.403013 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.402973 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-offline-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.403013 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.402992 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-ui-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.496027 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.495925 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:23:08.504034 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.503968 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.504034 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.504045 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80455e-f8e1-4e88-89e1-b6f057c5413c-feast-data\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.504079 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-online-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:08.504112 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.504143 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-offline-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:08.504179 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls podName:cb80455e-f8e1-4e88-89e1-b6f057c5413c nodeName:}" failed. No retries permitted until 2026-09-10 09:23:09.004160662 +0000 UTC m=+528.185849706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls") pod "feast-simple-feast-setup-6fc95c464f-rrr9g" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c") : secret "feast-simple-feast-setup-registry-tls" not found Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:08.504182 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.504211 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-ui-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:08.504234 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-online-tls podName:cb80455e-f8e1-4e88-89e1-b6f057c5413c nodeName:}" failed. No retries permitted until 2026-09-10 09:23:09.004218193 +0000 UTC m=+528.185907240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-online-tls") pod "feast-simple-feast-setup-6fc95c464f-rrr9g" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c") : secret "feast-simple-feast-setup-online-tls" not found Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:08.504259 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.504336 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dckbx\" (UniqueName: \"kubernetes.io/projected/cb80455e-f8e1-4e88-89e1-b6f057c5413c-kube-api-access-dckbx\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.504357 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:08.504359 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-ui-tls podName:cb80455e-f8e1-4e88-89e1-b6f057c5413c nodeName:}" failed. No retries permitted until 2026-09-10 09:23:09.004348809 +0000 UTC m=+528.186037854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-ui-tls") pod "feast-simple-feast-setup-6fc95c464f-rrr9g" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c") : secret "feast-simple-feast-setup-ui-tls" not found Sep 10 09:23:08.505065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.504738 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80455e-f8e1-4e88-89e1-b6f057c5413c-feast-data\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.506995 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.506968 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-offline-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:08.513346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:08.513258 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dckbx\" (UniqueName: \"kubernetes.io/projected/cb80455e-f8e1-4e88-89e1-b6f057c5413c-kube-api-access-dckbx\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:09.007119 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.007074 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:09.007660 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.007167 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-online-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:09.007660 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:09.007085 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 10 09:23:09.007660 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.007224 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-ui-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:09.007660 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:09.007243 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls podName:cb80455e-f8e1-4e88-89e1-b6f057c5413c nodeName:}" failed. No retries permitted until 2026-09-10 09:23:10.00722714 +0000 UTC m=+529.188916184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls") pod "feast-simple-feast-setup-6fc95c464f-rrr9g" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c") : secret "feast-simple-feast-setup-registry-tls" not found Sep 10 09:23:09.009902 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.009874 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-online-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:09.010053 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.009961 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-ui-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:09.540072 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.540037 2546 generic.go:356] "Generic (PLEG): container finished" podID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerID="b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3" exitCode=0 Sep 10 09:23:09.540072 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.540067 2546 generic.go:356] "Generic (PLEG): container finished" podID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerID="3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d" exitCode=0 Sep 10 09:23:09.540312 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.540128 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerDied","Data":"b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3"} Sep 10 09:23:09.540312 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.540179 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerDied","Data":"3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d"} Sep 10 09:23:09.732346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.732307 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:23:09.787711 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:09.787666 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:23:10.015459 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:10.015422 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:10.017912 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:10.017889 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls\") pod \"feast-simple-feast-setup-6fc95c464f-rrr9g\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:10.177104 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:10.177050 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:10.460293 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:10.460231 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g"] Sep 10 09:23:10.462562 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:23:10.462522 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb80455e_f8e1_4e88_89e1_b6f057c5413c.slice/crio-f6dc358610c02abe1a392e4029259a1f49a357fca9054926d57d51a4569cc40b WatchSource:0}: Error finding container f6dc358610c02abe1a392e4029259a1f49a357fca9054926d57d51a4569cc40b: Status 404 returned error can't find the container with id f6dc358610c02abe1a392e4029259a1f49a357fca9054926d57d51a4569cc40b Sep 10 09:23:10.545617 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:10.545580 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerStarted","Data":"b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4"} Sep 10 09:23:10.545785 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:10.545631 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerStarted","Data":"f6dc358610c02abe1a392e4029259a1f49a357fca9054926d57d51a4569cc40b"} Sep 10 09:23:14.655654 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:14.655408 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 10 09:23:15.288070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:15.288023 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 10 09:23:16.574442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:16.574403 2546 generic.go:356] "Generic (PLEG): container finished" podID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerID="b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4" exitCode=0 Sep 10 09:23:16.574835 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:16.574452 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerDied","Data":"b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4"} Sep 10 09:23:17.581569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:17.581530 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerStarted","Data":"0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680"} Sep 10 09:23:23.609002 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:23.608962 2546 generic.go:356] "Generic (PLEG): container finished" podID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerID="0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680" exitCode=0 Sep 10 09:23:23.609425 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:23.609037 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerDied","Data":"0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680"} Sep 10 09:23:24.620701 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:24.620649 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerStarted","Data":"6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1"} Sep 10 09:23:24.621203 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:24.620727 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerStarted","Data":"33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a"} Sep 10 09:23:24.621203 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:24.620750 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerStarted","Data":"98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99"} Sep 10 09:23:24.621203 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:24.620768 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerStarted","Data":"71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670"} Sep 10 09:23:24.650106 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:24.650029 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podStartSLOduration=16.649999598 podStartE2EDuration="16.649999598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:23:24.648016457 +0000 UTC m=+543.829705521" watchObservedRunningTime="2026-09-10 09:23:24.649999598 +0000 UTC m=+543.831688664" Sep 10 09:23:24.655454 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:24.655403 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 10 09:23:25.178115 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:25.178080 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:25.178421 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:25.178404 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:25.178538 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:25.178528 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:25.178762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:25.178748 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:25.180383 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:25.180330 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.23:6571: connect: connection refused" Sep 10 09:23:25.180383 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:25.180347 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Sep 10 09:23:25.180573 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:25.180346 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.23:8816: connect: connection refused" Sep 10 09:23:25.180573 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:25.180414 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Sep 10 09:23:25.289273 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:25.289213 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 10 09:23:28.177848 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:28.177793 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.23:8816: connect: connection refused" Sep 10 09:23:28.178390 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:28.177853 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Sep 10 09:23:28.178390 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:28.177803 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Sep 10 09:23:28.178390 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:28.177793 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.23:6571: connect: connection refused" Sep 10 09:23:30.177914 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:30.177846 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:30.177914 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:30.177922 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:30.178530 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:30.177945 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:30.178530 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:30.177966 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:31.178229 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:31.178175 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.23:6571: connect: connection refused" Sep 10 09:23:31.178892 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:31.178193 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Sep 10 09:23:31.178993 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:31.178239 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Sep 10 09:23:31.178993 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:31.178316 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.23:8816: connect: connection refused" Sep 10 09:23:34.178609 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.178558 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:34.179141 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.178757 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:34.179141 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.178949 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:34.179681 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.179655 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:34.179794 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.179712 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:34.179832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.179797 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:34.184498 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.184475 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:34.654941 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.654896 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Sep 10 09:23:34.655150 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.655045 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:23:34.683313 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:34.683075 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:23:35.032394 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:35.032351 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:23:35.288504 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:35.288404 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Sep 10 09:23:35.288879 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:35.288543 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:23:38.617581 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.617549 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:23:38.698070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.698030 2546 generic.go:356] "Generic (PLEG): container finished" podID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerID="931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506" exitCode=137 Sep 10 09:23:38.698070 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.698063 2546 generic.go:356] "Generic (PLEG): container finished" podID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerID="613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b" exitCode=137 Sep 10 09:23:38.698336 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.698102 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" Sep 10 09:23:38.698336 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.698104 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerDied","Data":"931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506"} Sep 10 09:23:38.698336 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.698163 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerDied","Data":"613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b"} Sep 10 09:23:38.698336 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.698181 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx" event={"ID":"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8","Type":"ContainerDied","Data":"08661ee23ebc8292b77db1c2336e7e8d0b771e227afcdc72c11512eeb707212d"} Sep 10 09:23:38.698336 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.698201 2546 scope.go:117] "RemoveContainer" containerID="b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3" Sep 10 09:23:38.708028 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.707992 2546 scope.go:117] "RemoveContainer" containerID="931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506" Sep 10 09:23:38.717036 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.717014 2546 scope.go:117] "RemoveContainer" containerID="3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d" Sep 10 09:23:38.725610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.725582 2546 scope.go:117] "RemoveContainer" containerID="613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b" Sep 10 09:23:38.734060 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.734036 2546 scope.go:117] "RemoveContainer" containerID="74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac" Sep 10 09:23:38.742513 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.742489 2546 scope.go:117] "RemoveContainer" containerID="c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557" Sep 10 09:23:38.750773 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.750749 2546 scope.go:117] "RemoveContainer" containerID="b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3" Sep 10 09:23:38.751176 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:38.751155 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3\": container with ID starting with b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3 not found: ID does not exist" containerID="b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3" Sep 10 09:23:38.751245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.751194 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3"} err="failed to get container status \"b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3\": rpc error: code = NotFound desc = could not find container \"b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3\": container with ID starting with b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3 not found: ID does not exist" Sep 10 09:23:38.751245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.751223 2546 scope.go:117] "RemoveContainer" containerID="931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506" Sep 10 09:23:38.751555 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:38.751533 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506\": container with ID starting with 931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506 not found: ID does not exist" containerID="931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506" Sep 10 09:23:38.751599 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.751563 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506"} err="failed to get container status \"931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506\": rpc error: code = NotFound desc = could not find container \"931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506\": container with ID starting with 931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506 not found: ID does not exist" Sep 10 09:23:38.751599 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.751581 2546 scope.go:117] "RemoveContainer" containerID="3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d" Sep 10 09:23:38.751821 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:38.751806 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d\": container with ID starting with 3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d not found: ID does not exist" containerID="3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d" Sep 10 09:23:38.751860 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.751824 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d"} err="failed to get container status \"3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d\": rpc error: code = NotFound desc = could not find container \"3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d\": container with ID starting with 3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d not found: ID does not exist" Sep 10 09:23:38.751860 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.751838 2546 scope.go:117] "RemoveContainer" containerID="613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b" Sep 10 09:23:38.752111 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:38.752095 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b\": container with ID starting with 613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b not found: ID does not exist" containerID="613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b" Sep 10 09:23:38.752176 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.752119 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b"} err="failed to get container status \"613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b\": rpc error: code = NotFound desc = could not find container \"613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b\": container with ID starting with 613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b not found: ID does not exist" Sep 10 09:23:38.752176 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.752139 2546 scope.go:117] "RemoveContainer" containerID="74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac" Sep 10 09:23:38.752394 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:38.752375 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac\": container with ID starting with 74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac not found: ID does not exist" containerID="74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac" Sep 10 09:23:38.752442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.752400 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac"} err="failed to get container status \"74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac\": rpc error: code = NotFound desc = could not find container \"74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac\": container with ID starting with 74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac not found: ID does not exist" Sep 10 09:23:38.752442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.752414 2546 scope.go:117] "RemoveContainer" containerID="c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557" Sep 10 09:23:38.752661 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:23:38.752644 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557\": container with ID starting with c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557 not found: ID does not exist" containerID="c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557" Sep 10 09:23:38.752699 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.752666 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557"} err="failed to get container status \"c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557\": rpc error: code = NotFound desc = could not find container \"c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557\": container with ID starting with c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557 not found: ID does not exist" Sep 10 09:23:38.752699 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.752679 2546 scope.go:117] "RemoveContainer" containerID="b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3" Sep 10 09:23:38.752884 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.752868 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3"} err="failed to get container status \"b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3\": rpc error: code = NotFound desc = could not find container \"b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3\": container with ID starting with b90b7c81e81611c76cf5f6fa00fe0d624be0254573504022620537fb789167c3 not found: ID does not exist" Sep 10 09:23:38.752921 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.752884 2546 scope.go:117] "RemoveContainer" containerID="931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506" Sep 10 09:23:38.753121 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.753102 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506"} err="failed to get container status \"931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506\": rpc error: code = NotFound desc = could not find container \"931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506\": container with ID starting with 931af41f2a612911476455745b1817e7582f27374955a7878688dbccca472506 not found: ID does not exist" Sep 10 09:23:38.753121 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.753120 2546 scope.go:117] "RemoveContainer" containerID="3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d" Sep 10 09:23:38.753320 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.753303 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d"} err="failed to get container status \"3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d\": rpc error: code = NotFound desc = could not find container \"3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d\": container with ID starting with 3898173ec512f6b230cf46bd5b5dfa83f5f9f3cdfae36770e772a243794d1f4d not found: ID does not exist" Sep 10 09:23:38.753360 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.753319 2546 scope.go:117] "RemoveContainer" containerID="613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b" Sep 10 09:23:38.753545 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.753527 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b"} err="failed to get container status \"613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b\": rpc error: code = NotFound desc = could not find container \"613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b\": container with ID starting with 613e12c69f8b7bf595e9ae74a8d1dfdd5b0ccb5e3c6d5dbb4e2c11ebb3f74a1b not found: ID does not exist" Sep 10 09:23:38.753589 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.753546 2546 scope.go:117] "RemoveContainer" containerID="74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac" Sep 10 09:23:38.753782 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.753763 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac"} err="failed to get container status \"74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac\": rpc error: code = NotFound desc = could not find container \"74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac\": container with ID starting with 74e624c8c882de18e1cecc06b1ac1559879d4127d817b3025476849ce3a672ac not found: ID does not exist" Sep 10 09:23:38.753841 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.753783 2546 scope.go:117] "RemoveContainer" containerID="c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557" Sep 10 09:23:38.754006 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.753992 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557"} err="failed to get container status \"c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557\": rpc error: code = NotFound desc = could not find container \"c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557\": container with ID starting with c4f1231b3e0807ba20acc9d209bddb3886bc6480908bffc87eca70e659f87557 not found: ID does not exist" Sep 10 09:23:38.796244 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.796212 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-registry-tls\") pod \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " Sep 10 09:23:38.796443 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.796266 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-offline-tls\") pod \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " Sep 10 09:23:38.796443 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.796317 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxnqn\" (UniqueName: \"kubernetes.io/projected/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-kube-api-access-rxnqn\") pod \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " Sep 10 09:23:38.796443 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.796362 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-feast-data\") pod \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " Sep 10 09:23:38.796443 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.796412 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-online-tls\") pod \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " Sep 10 09:23:38.796618 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.796541 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-ui-tls\") pod \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\" (UID: \"e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8\") " Sep 10 09:23:38.797023 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.796984 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-feast-data" (OuterVolumeSpecName: "feast-data") pod "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" (UID: "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:23:38.798828 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.798798 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" (UID: "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:23:38.799107 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.799087 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" (UID: "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:23:38.799176 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.799140 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" (UID: "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:23:38.799176 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.799150 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-online-tls" (OuterVolumeSpecName: "online-tls") pod "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" (UID: "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:23:38.799260 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.799235 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-kube-api-access-rxnqn" (OuterVolumeSpecName: "kube-api-access-rxnqn") pod "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" (UID: "e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8"). InnerVolumeSpecName "kube-api-access-rxnqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:23:38.897874 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.897836 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:23:38.897874 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.897874 2546 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-ui-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:23:38.897874 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.897883 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:23:38.898151 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.897891 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:23:38.898151 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.897899 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxnqn\" (UniqueName: \"kubernetes.io/projected/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-kube-api-access-rxnqn\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:23:38.898151 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:38.897907 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:23:39.023736 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:39.023696 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx"] Sep 10 09:23:39.027813 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:39.027779 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-wj2xx"] Sep 10 09:23:39.261127 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:23:39.261045 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" path="/var/lib/kubelet/pods/e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8/volumes" Sep 10 09:24:36.842363 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:36.842317 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:24:37.503295 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503246 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj"] Sep 10 09:24:37.503598 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503584 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="feast-init" Sep 10 09:24:37.503648 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503601 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="feast-init" Sep 10 09:24:37.503648 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503611 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="registry" Sep 10 09:24:37.503648 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503618 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="registry" Sep 10 09:24:37.503648 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503626 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="offline" Sep 10 09:24:37.503648 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503632 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="offline" Sep 10 09:24:37.503648 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503640 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="online" Sep 10 09:24:37.503648 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503645 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="online" Sep 10 09:24:37.503853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503658 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="ui" Sep 10 09:24:37.503853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503663 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="ui" Sep 10 09:24:37.503853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503672 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="feast-apply" Sep 10 09:24:37.503853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503677 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="feast-apply" Sep 10 09:24:37.503853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503726 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="registry" Sep 10 09:24:37.503853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503733 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="offline" Sep 10 09:24:37.503853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503740 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="ui" Sep 10 09:24:37.503853 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.503745 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16c486a-ff9f-4ffd-aceb-d6a1fe532fd8" containerName="online" Sep 10 09:24:37.508190 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.508163 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.518948 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.518918 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 10 09:24:37.519132 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.518926 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 10 09:24:37.519132 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.518974 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-bgzhc\"" Sep 10 09:24:37.519132 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.518976 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 10 09:24:37.520198 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.520178 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 10 09:24:37.520294 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.520178 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 10 09:24:37.521399 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.521381 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 10 09:24:37.521399 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.521391 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 10 09:24:37.543121 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.543085 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj"] Sep 10 09:24:37.604588 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.604538 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-offline-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.604588 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.604586 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f99a7b44-80c8-401a-8e5a-994884186d62-feast-data\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.604856 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.604626 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxcw\" (UniqueName: \"kubernetes.io/projected/f99a7b44-80c8-401a-8e5a-994884186d62-kube-api-access-psxcw\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.604856 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.604651 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f99a7b44-80c8-401a-8e5a-994884186d62-registry-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.604856 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.604674 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-ui-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.604856 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.604750 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-online-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.705624 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.705577 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxcw\" (UniqueName: \"kubernetes.io/projected/f99a7b44-80c8-401a-8e5a-994884186d62-kube-api-access-psxcw\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.705624 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.705625 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f99a7b44-80c8-401a-8e5a-994884186d62-registry-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.705910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.705648 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-ui-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.705910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.705673 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-online-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.705910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.705730 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-offline-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.705910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.705747 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f99a7b44-80c8-401a-8e5a-994884186d62-feast-data\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.705910 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:24:37.705853 2546 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 10 09:24:37.705910 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:24:37.705874 2546 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 10 09:24:37.706143 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:24:37.705924 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-online-tls podName:f99a7b44-80c8-401a-8e5a-994884186d62 nodeName:}" failed. No retries permitted until 2026-09-10 09:24:38.205900735 +0000 UTC m=+617.387589790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-online-tls") pod "feast-simple-feast-remote-setup-544bc66c4-dbvjj" (UID: "f99a7b44-80c8-401a-8e5a-994884186d62") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 10 09:24:37.706143 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:24:37.705955 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-offline-tls podName:f99a7b44-80c8-401a-8e5a-994884186d62 nodeName:}" failed. No retries permitted until 2026-09-10 09:24:38.205936289 +0000 UTC m=+617.387625339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-offline-tls") pod "feast-simple-feast-remote-setup-544bc66c4-dbvjj" (UID: "f99a7b44-80c8-401a-8e5a-994884186d62") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 10 09:24:37.706266 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.706245 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f99a7b44-80c8-401a-8e5a-994884186d62-feast-data\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.706425 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.706405 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f99a7b44-80c8-401a-8e5a-994884186d62-registry-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.708233 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.708216 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-ui-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:37.724177 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:37.724137 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxcw\" (UniqueName: \"kubernetes.io/projected/f99a7b44-80c8-401a-8e5a-994884186d62-kube-api-access-psxcw\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:38.209882 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:38.209831 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-online-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:38.210355 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:38.209909 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-offline-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:38.212530 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:38.212499 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-online-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:38.212682 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:38.212538 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-offline-tls\") pod \"feast-simple-feast-remote-setup-544bc66c4-dbvjj\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:38.417444 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:38.417405 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:24:38.522654 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:38.522615 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:24:38.713257 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:38.713199 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj"] Sep 10 09:24:38.718127 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:24:38.718007 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf99a7b44_80c8_401a_8e5a_994884186d62.slice/crio-72af7ebbab389d6a7cf44499ebe01ccdb77718a2a923df6cc289044a2c9a3ec3 WatchSource:0}: Error finding container 72af7ebbab389d6a7cf44499ebe01ccdb77718a2a923df6cc289044a2c9a3ec3: Status 404 returned error can't find the container with id 72af7ebbab389d6a7cf44499ebe01ccdb77718a2a923df6cc289044a2c9a3ec3 Sep 10 09:24:38.949381 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:38.949339 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerStarted","Data":"121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89"} Sep 10 09:24:38.949381 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:38.949388 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerStarted","Data":"72af7ebbab389d6a7cf44499ebe01ccdb77718a2a923df6cc289044a2c9a3ec3"} Sep 10 09:24:44.979899 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:44.979863 2546 generic.go:356] "Generic (PLEG): container finished" podID="f99a7b44-80c8-401a-8e5a-994884186d62" containerID="121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89" exitCode=0 Sep 10 09:24:44.980332 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:44.979913 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerDied","Data":"121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89"} Sep 10 09:24:45.536875 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:45.536828 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:24:45.985801 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:45.985762 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerStarted","Data":"36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2"} Sep 10 09:24:49.754937 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:24:49.754889 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:25:02.278641 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:02.278606 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:25:02.575578 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:02.575484 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:25:10.088936 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:10.088898 2546 generic.go:356] "Generic (PLEG): container finished" podID="f99a7b44-80c8-401a-8e5a-994884186d62" containerID="36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2" exitCode=0 Sep 10 09:25:10.089365 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:10.088945 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerDied","Data":"36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2"} Sep 10 09:25:11.098177 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.098124 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerStarted","Data":"b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c"} Sep 10 09:25:11.098721 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.098188 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerStarted","Data":"0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70"} Sep 10 09:25:11.098721 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.098202 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerStarted","Data":"3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979"} Sep 10 09:25:11.122813 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.122729 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podStartSLOduration=34.122706805 podStartE2EDuration="34.122706805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:25:11.122008029 +0000 UTC m=+650.303697106" watchObservedRunningTime="2026-09-10 09:25:11.122706805 +0000 UTC m=+650.304395872" Sep 10 09:25:11.418310 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.418183 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:11.418310 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.418252 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:11.418310 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.418269 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:11.420383 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.420335 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Sep 10 09:25:11.420555 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.420395 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 10 09:25:11.420688 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:11.420642 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 10 09:25:14.418560 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:14.418512 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Sep 10 09:25:14.418560 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:14.418520 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 10 09:25:14.419129 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:14.418512 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 10 09:25:17.418667 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:17.418620 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 10 09:25:17.419181 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:17.418683 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 10 09:25:17.419181 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:17.418628 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Sep 10 09:25:18.418021 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:18.417972 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:18.418210 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:18.418042 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:18.418210 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:18.418058 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:20.419156 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:20.419116 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:20.419618 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:20.419399 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:20.419951 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:20.419931 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:20.420044 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:20.419980 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:20.449757 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:20.449725 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:21.169268 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:21.169231 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:25:21.501575 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:25:21.501541 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:26:23.623616 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.623408 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:26:23.634074 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.634033 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj"] Sep 10 09:26:23.634316 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.634275 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" containerID="cri-o://3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979" gracePeriod=30 Sep 10 09:26:23.634432 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.634397 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" containerID="cri-o://b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c" gracePeriod=30 Sep 10 09:26:23.634492 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.634415 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="offline" containerID="cri-o://0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70" gracePeriod=30 Sep 10 09:26:23.724390 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.724345 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:26:23.737532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.737495 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g"] Sep 10 09:26:23.737762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.737730 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="registry" containerID="cri-o://71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670" gracePeriod=30 Sep 10 09:26:23.737832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.737779 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="offline" containerID="cri-o://33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a" gracePeriod=30 Sep 10 09:26:23.737832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.737803 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" containerID="cri-o://6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1" gracePeriod=30 Sep 10 09:26:23.737942 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.737804 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" containerID="cri-o://98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99" gracePeriod=30 Sep 10 09:26:23.825011 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.824973 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:26:23.878241 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:23.878146 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:26:24.180122 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:24.179971 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Sep 10 09:26:24.680523 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:24.680455 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Sep 10 09:26:25.417035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:25.416998 2546 generic.go:356] "Generic (PLEG): container finished" podID="f99a7b44-80c8-401a-8e5a-994884186d62" containerID="b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c" exitCode=0 Sep 10 09:26:25.417035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:25.417027 2546 generic.go:356] "Generic (PLEG): container finished" podID="f99a7b44-80c8-401a-8e5a-994884186d62" containerID="3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979" exitCode=0 Sep 10 09:26:25.417265 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:25.417069 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerDied","Data":"b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c"} Sep 10 09:26:25.417265 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:25.417111 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerDied","Data":"3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979"} Sep 10 09:26:25.420228 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:25.420196 2546 generic.go:356] "Generic (PLEG): container finished" podID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerID="6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1" exitCode=0 Sep 10 09:26:25.420228 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:25.420221 2546 generic.go:356] "Generic (PLEG): container finished" podID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerID="98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99" exitCode=0 Sep 10 09:26:25.420446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:25.420269 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerDied","Data":"6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1"} Sep 10 09:26:25.420446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:25.420336 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerDied","Data":"98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99"} Sep 10 09:26:28.900426 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:28.900387 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:26:30.419671 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:30.419621 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 10 09:26:31.148944 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:31.148898 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 10 09:26:34.179867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:34.179812 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Sep 10 09:26:34.680173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:34.680121 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Sep 10 09:26:40.420109 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:40.420048 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 10 09:26:41.149502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:41.149456 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 10 09:26:44.179990 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:44.179912 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Sep 10 09:26:44.180519 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:44.180109 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:26:44.680085 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:44.680037 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Sep 10 09:26:44.680316 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:44.680161 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:26:50.420650 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:50.420605 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 10 09:26:50.421122 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:50.420731 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:26:51.149741 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:51.149694 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 10 09:26:51.149959 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:51.149832 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:26:54.180494 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.180442 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.23:8816: connect: connection refused" Sep 10 09:26:54.180951 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.180443 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Sep 10 09:26:54.180951 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.180449 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.23:6571: connect: connection refused" Sep 10 09:26:54.302588 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.302554 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:26:54.382773 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.382739 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:26:54.420960 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.420853 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-online-tls\") pod \"f99a7b44-80c8-401a-8e5a-994884186d62\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " Sep 10 09:26:54.420960 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.420928 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f99a7b44-80c8-401a-8e5a-994884186d62-registry-tls\") pod \"f99a7b44-80c8-401a-8e5a-994884186d62\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " Sep 10 09:26:54.421179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.421040 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-offline-tls\") pod \"f99a7b44-80c8-401a-8e5a-994884186d62\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " Sep 10 09:26:54.421179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.421069 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-ui-tls\") pod \"f99a7b44-80c8-401a-8e5a-994884186d62\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " Sep 10 09:26:54.421179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.421097 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f99a7b44-80c8-401a-8e5a-994884186d62-feast-data\") pod \"f99a7b44-80c8-401a-8e5a-994884186d62\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " Sep 10 09:26:54.421179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.421142 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psxcw\" (UniqueName: \"kubernetes.io/projected/f99a7b44-80c8-401a-8e5a-994884186d62-kube-api-access-psxcw\") pod \"f99a7b44-80c8-401a-8e5a-994884186d62\" (UID: \"f99a7b44-80c8-401a-8e5a-994884186d62\") " Sep 10 09:26:54.421439 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.421267 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99a7b44-80c8-401a-8e5a-994884186d62-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f99a7b44-80c8-401a-8e5a-994884186d62" (UID: "f99a7b44-80c8-401a-8e5a-994884186d62"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:26:54.421439 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.421414 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f99a7b44-80c8-401a-8e5a-994884186d62-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.422071 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.422041 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99a7b44-80c8-401a-8e5a-994884186d62-feast-data" (OuterVolumeSpecName: "feast-data") pod "f99a7b44-80c8-401a-8e5a-994884186d62" (UID: "f99a7b44-80c8-401a-8e5a-994884186d62"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:26:54.423511 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.423467 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99a7b44-80c8-401a-8e5a-994884186d62-kube-api-access-psxcw" (OuterVolumeSpecName: "kube-api-access-psxcw") pod "f99a7b44-80c8-401a-8e5a-994884186d62" (UID: "f99a7b44-80c8-401a-8e5a-994884186d62"). InnerVolumeSpecName "kube-api-access-psxcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:26:54.423675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.423647 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f99a7b44-80c8-401a-8e5a-994884186d62" (UID: "f99a7b44-80c8-401a-8e5a-994884186d62"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:26:54.423942 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.423908 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f99a7b44-80c8-401a-8e5a-994884186d62" (UID: "f99a7b44-80c8-401a-8e5a-994884186d62"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:26:54.424072 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.424050 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-online-tls" (OuterVolumeSpecName: "online-tls") pod "f99a7b44-80c8-401a-8e5a-994884186d62" (UID: "f99a7b44-80c8-401a-8e5a-994884186d62"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:26:54.522038 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.521997 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-ui-tls\") pod \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " Sep 10 09:26:54.522038 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522050 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dckbx\" (UniqueName: \"kubernetes.io/projected/cb80455e-f8e1-4e88-89e1-b6f057c5413c-kube-api-access-dckbx\") pod \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " Sep 10 09:26:54.522449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522067 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls\") pod \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " Sep 10 09:26:54.522449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522090 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80455e-f8e1-4e88-89e1-b6f057c5413c-feast-data\") pod \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " Sep 10 09:26:54.522449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522152 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-online-tls\") pod \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " Sep 10 09:26:54.522449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522166 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-offline-tls\") pod \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\" (UID: \"cb80455e-f8e1-4e88-89e1-b6f057c5413c\") " Sep 10 09:26:54.522449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522392 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.522449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522407 2546 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-ui-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.522449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522415 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f99a7b44-80c8-401a-8e5a-994884186d62-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.522449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522423 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxcw\" (UniqueName: \"kubernetes.io/projected/f99a7b44-80c8-401a-8e5a-994884186d62-kube-api-access-psxcw\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.522449 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522456 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f99a7b44-80c8-401a-8e5a-994884186d62-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.522955 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.522769 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb80455e-f8e1-4e88-89e1-b6f057c5413c-feast-data" (OuterVolumeSpecName: "feast-data") pod "cb80455e-f8e1-4e88-89e1-b6f057c5413c" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:26:54.524485 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.524452 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cb80455e-f8e1-4e88-89e1-b6f057c5413c" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:26:54.524738 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.524723 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-online-tls" (OuterVolumeSpecName: "online-tls") pod "cb80455e-f8e1-4e88-89e1-b6f057c5413c" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:26:54.524881 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.524855 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "cb80455e-f8e1-4e88-89e1-b6f057c5413c" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:26:54.524928 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.524907 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb80455e-f8e1-4e88-89e1-b6f057c5413c-kube-api-access-dckbx" (OuterVolumeSpecName: "kube-api-access-dckbx") pod "cb80455e-f8e1-4e88-89e1-b6f057c5413c" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c"). InnerVolumeSpecName "kube-api-access-dckbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:26:54.524928 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.524909 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "cb80455e-f8e1-4e88-89e1-b6f057c5413c" (UID: "cb80455e-f8e1-4e88-89e1-b6f057c5413c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:26:54.542066 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.542032 2546 generic.go:356] "Generic (PLEG): container finished" podID="f99a7b44-80c8-401a-8e5a-994884186d62" containerID="0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70" exitCode=137 Sep 10 09:26:54.542396 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.542095 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" Sep 10 09:26:54.542396 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.542117 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerDied","Data":"0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70"} Sep 10 09:26:54.542396 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.542173 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj" event={"ID":"f99a7b44-80c8-401a-8e5a-994884186d62","Type":"ContainerDied","Data":"72af7ebbab389d6a7cf44499ebe01ccdb77718a2a923df6cc289044a2c9a3ec3"} Sep 10 09:26:54.542396 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.542194 2546 scope.go:117] "RemoveContainer" containerID="b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c" Sep 10 09:26:54.545437 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.545404 2546 generic.go:356] "Generic (PLEG): container finished" podID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerID="33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a" exitCode=137 Sep 10 09:26:54.545437 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.545433 2546 generic.go:356] "Generic (PLEG): container finished" podID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerID="71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670" exitCode=137 Sep 10 09:26:54.545610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.545471 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" Sep 10 09:26:54.545610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.545471 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerDied","Data":"33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a"} Sep 10 09:26:54.545610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.545523 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerDied","Data":"71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670"} Sep 10 09:26:54.545610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.545547 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g" event={"ID":"cb80455e-f8e1-4e88-89e1-b6f057c5413c","Type":"ContainerDied","Data":"f6dc358610c02abe1a392e4029259a1f49a357fca9054926d57d51a4569cc40b"} Sep 10 09:26:54.551840 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.551818 2546 scope.go:117] "RemoveContainer" containerID="0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70" Sep 10 09:26:54.561213 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.561192 2546 scope.go:117] "RemoveContainer" containerID="3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979" Sep 10 09:26:54.565960 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.565932 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj"] Sep 10 09:26:54.570523 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.570500 2546 scope.go:117] "RemoveContainer" containerID="36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2" Sep 10 09:26:54.573699 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.573669 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-544bc66c4-dbvjj"] Sep 10 09:26:54.580076 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.580051 2546 scope.go:117] "RemoveContainer" containerID="121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89" Sep 10 09:26:54.586416 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.586385 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g"] Sep 10 09:26:54.589676 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.589637 2546 scope.go:117] "RemoveContainer" containerID="b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c" Sep 10 09:26:54.590054 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.590032 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c\": container with ID starting with b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c not found: ID does not exist" containerID="b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c" Sep 10 09:26:54.590156 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.590063 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c"} err="failed to get container status \"b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c\": rpc error: code = NotFound desc = could not find container \"b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c\": container with ID starting with b6de959a20443bc1dbafa161ac640ab52695908b1792f6de9bf880c91ee6ab2c not found: ID does not exist" Sep 10 09:26:54.590156 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.590083 2546 scope.go:117] "RemoveContainer" containerID="0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70" Sep 10 09:26:54.590450 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.590431 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70\": container with ID starting with 0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70 not found: ID does not exist" containerID="0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70" Sep 10 09:26:54.590532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.590457 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70"} err="failed to get container status \"0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70\": rpc error: code = NotFound desc = could not find container \"0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70\": container with ID starting with 0a4e1847479956042d9958af8693ff696889a057a63c30dab5d399322589fb70 not found: ID does not exist" Sep 10 09:26:54.590532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.590475 2546 scope.go:117] "RemoveContainer" containerID="3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979" Sep 10 09:26:54.590741 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.590719 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6fc95c464f-rrr9g"] Sep 10 09:26:54.590809 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.590723 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979\": container with ID starting with 3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979 not found: ID does not exist" containerID="3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979" Sep 10 09:26:54.590863 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.590797 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979"} err="failed to get container status \"3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979\": rpc error: code = NotFound desc = could not find container \"3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979\": container with ID starting with 3b2191ec26f6e4cdb8288e713982f0d5b81c523c7c5b26b9d0af6d5242448979 not found: ID does not exist" Sep 10 09:26:54.590863 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.590820 2546 scope.go:117] "RemoveContainer" containerID="36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2" Sep 10 09:26:54.591118 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.591097 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2\": container with ID starting with 36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2 not found: ID does not exist" containerID="36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2" Sep 10 09:26:54.591179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.591124 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2"} err="failed to get container status \"36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2\": rpc error: code = NotFound desc = could not find container \"36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2\": container with ID starting with 36c9257475a4abfac2cda8204acfbd44c5aeee0b02d4e28cb6e7c0774d1d36f2 not found: ID does not exist" Sep 10 09:26:54.591179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.591138 2546 scope.go:117] "RemoveContainer" containerID="121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89" Sep 10 09:26:54.591364 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.591341 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89\": container with ID starting with 121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89 not found: ID does not exist" containerID="121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89" Sep 10 09:26:54.591418 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.591370 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89"} err="failed to get container status \"121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89\": rpc error: code = NotFound desc = could not find container \"121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89\": container with ID starting with 121f77cad5d942ecfd27c0804119cc301d0a7fabdce4648ed5e27e2d28a24b89 not found: ID does not exist" Sep 10 09:26:54.591418 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.591392 2546 scope.go:117] "RemoveContainer" containerID="6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1" Sep 10 09:26:54.599839 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.599817 2546 scope.go:117] "RemoveContainer" containerID="33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a" Sep 10 09:26:54.608355 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.608327 2546 scope.go:117] "RemoveContainer" containerID="98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99" Sep 10 09:26:54.616778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.616751 2546 scope.go:117] "RemoveContainer" containerID="71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670" Sep 10 09:26:54.623343 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.623318 2546 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-ui-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.623343 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.623343 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dckbx\" (UniqueName: \"kubernetes.io/projected/cb80455e-f8e1-4e88-89e1-b6f057c5413c-kube-api-access-dckbx\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.623502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.623353 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.623502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.623369 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cb80455e-f8e1-4e88-89e1-b6f057c5413c-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.623502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.623379 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.623502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.623386 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cb80455e-f8e1-4e88-89e1-b6f057c5413c-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:26:54.625016 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.624997 2546 scope.go:117] "RemoveContainer" containerID="0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680" Sep 10 09:26:54.633054 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.633032 2546 scope.go:117] "RemoveContainer" containerID="b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4" Sep 10 09:26:54.641382 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.641180 2546 scope.go:117] "RemoveContainer" containerID="6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1" Sep 10 09:26:54.641581 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.641562 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1\": container with ID starting with 6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1 not found: ID does not exist" containerID="6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1" Sep 10 09:26:54.641626 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.641590 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1"} err="failed to get container status \"6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1\": rpc error: code = NotFound desc = could not find container \"6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1\": container with ID starting with 6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1 not found: ID does not exist" Sep 10 09:26:54.641626 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.641612 2546 scope.go:117] "RemoveContainer" containerID="33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a" Sep 10 09:26:54.641892 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.641875 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a\": container with ID starting with 33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a not found: ID does not exist" containerID="33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a" Sep 10 09:26:54.641960 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.641895 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a"} err="failed to get container status \"33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a\": rpc error: code = NotFound desc = could not find container \"33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a\": container with ID starting with 33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a not found: ID does not exist" Sep 10 09:26:54.641960 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.641910 2546 scope.go:117] "RemoveContainer" containerID="98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99" Sep 10 09:26:54.642154 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.642138 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99\": container with ID starting with 98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99 not found: ID does not exist" containerID="98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99" Sep 10 09:26:54.642195 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.642157 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99"} err="failed to get container status \"98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99\": rpc error: code = NotFound desc = could not find container \"98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99\": container with ID starting with 98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99 not found: ID does not exist" Sep 10 09:26:54.642195 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.642170 2546 scope.go:117] "RemoveContainer" containerID="71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670" Sep 10 09:26:54.642461 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.642434 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670\": container with ID starting with 71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670 not found: ID does not exist" containerID="71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670" Sep 10 09:26:54.642548 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.642464 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670"} err="failed to get container status \"71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670\": rpc error: code = NotFound desc = could not find container \"71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670\": container with ID starting with 71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670 not found: ID does not exist" Sep 10 09:26:54.642548 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.642481 2546 scope.go:117] "RemoveContainer" containerID="0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680" Sep 10 09:26:54.642730 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.642714 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680\": container with ID starting with 0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680 not found: ID does not exist" containerID="0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680" Sep 10 09:26:54.642770 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.642734 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680"} err="failed to get container status \"0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680\": rpc error: code = NotFound desc = could not find container \"0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680\": container with ID starting with 0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680 not found: ID does not exist" Sep 10 09:26:54.642770 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.642748 2546 scope.go:117] "RemoveContainer" containerID="b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4" Sep 10 09:26:54.643042 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:26:54.643027 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4\": container with ID starting with b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4 not found: ID does not exist" containerID="b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4" Sep 10 09:26:54.643088 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.643046 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4"} err="failed to get container status \"b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4\": rpc error: code = NotFound desc = could not find container \"b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4\": container with ID starting with b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4 not found: ID does not exist" Sep 10 09:26:54.643088 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.643057 2546 scope.go:117] "RemoveContainer" containerID="6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1" Sep 10 09:26:54.643309 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.643272 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1"} err="failed to get container status \"6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1\": rpc error: code = NotFound desc = could not find container \"6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1\": container with ID starting with 6b0b71d37be074eb737772c58f46d7dd2bf4218683e2f3f0f42121063e8a1dd1 not found: ID does not exist" Sep 10 09:26:54.643309 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.643309 2546 scope.go:117] "RemoveContainer" containerID="33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a" Sep 10 09:26:54.643564 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.643542 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a"} err="failed to get container status \"33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a\": rpc error: code = NotFound desc = could not find container \"33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a\": container with ID starting with 33cee0677bbb689917b59902277aad7ee61964d114587969bca88efa43ed882a not found: ID does not exist" Sep 10 09:26:54.643641 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.643567 2546 scope.go:117] "RemoveContainer" containerID="98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99" Sep 10 09:26:54.643842 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.643818 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99"} err="failed to get container status \"98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99\": rpc error: code = NotFound desc = could not find container \"98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99\": container with ID starting with 98d644803979c03d8729b8fe19fee59310dcdba0594b700b64a3b69df6905b99 not found: ID does not exist" Sep 10 09:26:54.643905 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.643845 2546 scope.go:117] "RemoveContainer" containerID="71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670" Sep 10 09:26:54.644118 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.644101 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670"} err="failed to get container status \"71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670\": rpc error: code = NotFound desc = could not find container \"71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670\": container with ID starting with 71e0b9d84b1167dace8958b24bc524cc44aed966a1a87ac4a019345d56d1b670 not found: ID does not exist" Sep 10 09:26:54.644168 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.644121 2546 scope.go:117] "RemoveContainer" containerID="0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680" Sep 10 09:26:54.644391 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.644372 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680"} err="failed to get container status \"0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680\": rpc error: code = NotFound desc = could not find container \"0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680\": container with ID starting with 0c7b6f022a2dcc8f49d8a5095308973ec02ea0c7d0c168e9522dab9cf8283680 not found: ID does not exist" Sep 10 09:26:54.644472 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.644392 2546 scope.go:117] "RemoveContainer" containerID="b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4" Sep 10 09:26:54.644627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:54.644607 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4"} err="failed to get container status \"b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4\": rpc error: code = NotFound desc = could not find container \"b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4\": container with ID starting with b487cb79645d041ec58eb22dcc2e2e258f14b7436890e5ba38c46f6750bf92c4 not found: ID does not exist" Sep 10 09:26:55.260539 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:55.260493 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" path="/var/lib/kubelet/pods/cb80455e-f8e1-4e88-89e1-b6f057c5413c/volumes" Sep 10 09:26:55.261358 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:55.261333 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" path="/var/lib/kubelet/pods/f99a7b44-80c8-401a-8e5a-994884186d62/volumes" Sep 10 09:26:55.535469 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:55.535382 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:26:55.606530 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:26:55.606496 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:27:06.666940 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.666896 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:27:06.874206 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874163 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-jb5m6"] Sep 10 09:27:06.874493 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874478 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" Sep 10 09:27:06.874547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874495 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" Sep 10 09:27:06.874547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874505 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" Sep 10 09:27:06.874547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874511 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" Sep 10 09:27:06.874547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874520 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="feast-apply" Sep 10 09:27:06.874547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874526 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="feast-apply" Sep 10 09:27:06.874547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874540 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="registry" Sep 10 09:27:06.874547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874545 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="registry" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874552 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="offline" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874558 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="offline" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874564 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874569 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874576 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="feast-init" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874581 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="feast-init" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874587 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="offline" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874592 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="offline" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874598 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="feast-init" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874603 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="feast-init" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874609 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="feast-apply" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874614 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="feast-apply" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874619 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874624 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874663 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="registry" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874669 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="offline" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874675 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="online" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874680 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb80455e-f8e1-4e88-89e1-b6f057c5413c" containerName="ui" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874686 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="offline" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874696 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="online" Sep 10 09:27:06.874778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.874704 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f99a7b44-80c8-401a-8e5a-994884186d62" containerName="ui" Sep 10 09:27:06.879804 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.879767 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:27:06.883269 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.883204 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 10 09:27:06.883269 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.883227 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 10 09:27:06.883521 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.883301 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-2vz2k\"" Sep 10 09:27:06.884000 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.883981 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 10 09:27:06.886617 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.886591 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jb5m6"] Sep 10 09:27:06.904319 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.904265 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-rp5wz"] Sep 10 09:27:06.907541 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.907516 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-rp5wz" Sep 10 09:27:06.910459 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.910425 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:27:06.916002 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.915958 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rp5wz"] Sep 10 09:27:06.923377 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.923266 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/745dab2f-3808-4299-bac0-0fb99b44fcdd-postgresdata\") pod \"postgres-864846d4f5-jb5m6\" (UID: \"745dab2f-3808-4299-bac0-0fb99b44fcdd\") " pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:27:06.923512 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.923395 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmsh\" (UniqueName: \"kubernetes.io/projected/745dab2f-3808-4299-bac0-0fb99b44fcdd-kube-api-access-5tmsh\") pod \"postgres-864846d4f5-jb5m6\" (UID: \"745dab2f-3808-4299-bac0-0fb99b44fcdd\") " pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:27:06.923512 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.923448 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bft\" (UniqueName: \"kubernetes.io/projected/283dc883-6a74-4f82-bf45-d5f31b95bed4-kube-api-access-z2bft\") pod \"redis-69bbd798b6-rp5wz\" (UID: \"283dc883-6a74-4f82-bf45-d5f31b95bed4\") " pod="test-ns-feast/redis-69bbd798b6-rp5wz" Sep 10 09:27:06.984078 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:06.984041 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:27:07.024500 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.024459 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/745dab2f-3808-4299-bac0-0fb99b44fcdd-postgresdata\") pod \"postgres-864846d4f5-jb5m6\" (UID: \"745dab2f-3808-4299-bac0-0fb99b44fcdd\") " pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:27:07.024652 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.024506 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmsh\" (UniqueName: \"kubernetes.io/projected/745dab2f-3808-4299-bac0-0fb99b44fcdd-kube-api-access-5tmsh\") pod \"postgres-864846d4f5-jb5m6\" (UID: \"745dab2f-3808-4299-bac0-0fb99b44fcdd\") " pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:27:07.024652 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.024562 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bft\" (UniqueName: \"kubernetes.io/projected/283dc883-6a74-4f82-bf45-d5f31b95bed4-kube-api-access-z2bft\") pod \"redis-69bbd798b6-rp5wz\" (UID: \"283dc883-6a74-4f82-bf45-d5f31b95bed4\") " pod="test-ns-feast/redis-69bbd798b6-rp5wz" Sep 10 09:27:07.025055 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.025035 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/745dab2f-3808-4299-bac0-0fb99b44fcdd-postgresdata\") pod \"postgres-864846d4f5-jb5m6\" (UID: \"745dab2f-3808-4299-bac0-0fb99b44fcdd\") " pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:27:07.034707 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.034669 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmsh\" (UniqueName: \"kubernetes.io/projected/745dab2f-3808-4299-bac0-0fb99b44fcdd-kube-api-access-5tmsh\") pod \"postgres-864846d4f5-jb5m6\" (UID: \"745dab2f-3808-4299-bac0-0fb99b44fcdd\") " pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:27:07.034832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.034669 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bft\" (UniqueName: \"kubernetes.io/projected/283dc883-6a74-4f82-bf45-d5f31b95bed4-kube-api-access-z2bft\") pod \"redis-69bbd798b6-rp5wz\" (UID: \"283dc883-6a74-4f82-bf45-d5f31b95bed4\") " pod="test-ns-feast/redis-69bbd798b6-rp5wz" Sep 10 09:27:07.202031 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.201937 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:27:07.219215 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.219178 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-rp5wz" Sep 10 09:27:07.474896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.474857 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jb5m6"] Sep 10 09:27:07.477648 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:27:07.477617 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod745dab2f_3808_4299_bac0_0fb99b44fcdd.slice/crio-a57ea309301dc0f7d5ae41d355cf938623ee0dc88afa5a35381a33c9f213290d WatchSource:0}: Error finding container a57ea309301dc0f7d5ae41d355cf938623ee0dc88afa5a35381a33c9f213290d: Status 404 returned error can't find the container with id a57ea309301dc0f7d5ae41d355cf938623ee0dc88afa5a35381a33c9f213290d Sep 10 09:27:07.479674 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.479655 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 09:27:07.504268 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.504231 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rp5wz"] Sep 10 09:27:07.507770 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:27:07.507742 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod283dc883_6a74_4f82_bf45_d5f31b95bed4.slice/crio-8740f11a3a50b616854aab45dc4aba94c164a41ecfce5c51fea780454434344c WatchSource:0}: Error finding container 8740f11a3a50b616854aab45dc4aba94c164a41ecfce5c51fea780454434344c: Status 404 returned error can't find the container with id 8740f11a3a50b616854aab45dc4aba94c164a41ecfce5c51fea780454434344c Sep 10 09:27:07.600796 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.600760 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-jb5m6" event={"ID":"745dab2f-3808-4299-bac0-0fb99b44fcdd","Type":"ContainerStarted","Data":"a57ea309301dc0f7d5ae41d355cf938623ee0dc88afa5a35381a33c9f213290d"} Sep 10 09:27:07.601899 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:07.601866 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-rp5wz" event={"ID":"283dc883-6a74-4f82-bf45-d5f31b95bed4","Type":"ContainerStarted","Data":"8740f11a3a50b616854aab45dc4aba94c164a41ecfce5c51fea780454434344c"} Sep 10 09:27:09.613972 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:09.613932 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-rp5wz" event={"ID":"283dc883-6a74-4f82-bf45-d5f31b95bed4","Type":"ContainerStarted","Data":"c790e8d2e189e5582bd189fdbde56605993aa19fa805058c0192a58e6a556eff"} Sep 10 09:27:09.627694 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:09.627631 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-rp5wz" podStartSLOduration=2.024398458 podStartE2EDuration="3.627607348s" totalImagesPullingTime="1.60320889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:27:09.62746752 +0000 UTC m=+768.809156587" watchObservedRunningTime="2026-09-10 09:27:09.627607348 +0000 UTC m=+768.809296457" Sep 10 09:27:13.632727 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:13.632687 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-jb5m6" event={"ID":"745dab2f-3808-4299-bac0-0fb99b44fcdd","Type":"ContainerStarted","Data":"144e89ad1e29569f39c08743a88a5a027b3c25a1e6cb5fa7a8174357e2c841d3"} Sep 10 09:27:13.650199 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:13.650143 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-jb5m6" podStartSLOduration=2.447689061 podStartE2EDuration="7.650114324s" totalImagesPullingTime="5.202425263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:27:13.649058953 +0000 UTC m=+772.830748032" watchObservedRunningTime="2026-09-10 09:27:13.650114324 +0000 UTC m=+772.831803390" Sep 10 09:27:17.613640 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.613603 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:27:17.712150 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.712105 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:27:17.716540 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.716509 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9"] Sep 10 09:27:17.720181 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.720155 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.722569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.722540 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 10 09:27:17.722727 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.722572 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-v5r7c\"" Sep 10 09:27:17.722727 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.722540 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 10 09:27:17.723009 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.722993 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 10 09:27:17.723061 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.722997 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 10 09:27:17.733838 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.733806 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9"] Sep 10 09:27:17.775488 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.775443 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:27:17.825855 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.825813 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvdt9\" (UniqueName: \"kubernetes.io/projected/33375e36-23ce-46e0-bd58-1b7f8ed334db-kube-api-access-fvdt9\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.825855 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.825858 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33375e36-23ce-46e0-bd58-1b7f8ed334db-feast-data\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.826147 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.825934 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-offline-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.826147 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.825996 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-registry-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.826147 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.826054 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-online-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.857492 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.857453 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:27:17.926762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.926659 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-offline-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.926762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.926723 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-registry-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.926762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.926754 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-online-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.927006 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.926774 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvdt9\" (UniqueName: \"kubernetes.io/projected/33375e36-23ce-46e0-bd58-1b7f8ed334db-kube-api-access-fvdt9\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.927006 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.926797 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33375e36-23ce-46e0-bd58-1b7f8ed334db-feast-data\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.927006 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:27:17.926918 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 10 09:27:17.927006 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:27:17.927002 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-registry-tls podName:33375e36-23ce-46e0-bd58-1b7f8ed334db nodeName:}" failed. No retries permitted until 2026-09-10 09:27:18.426979116 +0000 UTC m=+777.608668172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-registry-tls") pod "feast-credit-scoring-6b8dd4655f-qdvm9" (UID: "33375e36-23ce-46e0-bd58-1b7f8ed334db") : secret "feast-credit-scoring-registry-tls" not found Sep 10 09:27:17.927253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.927231 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33375e36-23ce-46e0-bd58-1b7f8ed334db-feast-data\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.929346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.929319 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-online-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.929448 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.929374 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-offline-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:17.935740 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:17.935713 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvdt9\" (UniqueName: \"kubernetes.io/projected/33375e36-23ce-46e0-bd58-1b7f8ed334db-kube-api-access-fvdt9\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:18.077571 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:18.077501 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:27:18.431631 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:18.431589 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-registry-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:18.434300 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:18.434261 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-registry-tls\") pod \"feast-credit-scoring-6b8dd4655f-qdvm9\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:18.630695 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:18.630644 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:18.911577 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:18.911552 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9"] Sep 10 09:27:18.913851 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:27:18.913820 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33375e36_23ce_46e0_bd58_1b7f8ed334db.slice/crio-e84101e4e5f02839db304b3d67164b486bc596b8d024fbd3e148705646b424ab WatchSource:0}: Error finding container e84101e4e5f02839db304b3d67164b486bc596b8d024fbd3e148705646b424ab: Status 404 returned error can't find the container with id e84101e4e5f02839db304b3d67164b486bc596b8d024fbd3e148705646b424ab Sep 10 09:27:19.656506 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:19.656462 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerStarted","Data":"db8ddc07600dc88ccb3ff6207804329f8004f1f86c70c1dc17b64f227bc84169"} Sep 10 09:27:19.656910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:19.656570 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerStarted","Data":"e84101e4e5f02839db304b3d67164b486bc596b8d024fbd3e148705646b424ab"} Sep 10 09:27:20.662604 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:20.662561 2546 generic.go:356] "Generic (PLEG): container finished" podID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerID="db8ddc07600dc88ccb3ff6207804329f8004f1f86c70c1dc17b64f227bc84169" exitCode=0 Sep 10 09:27:20.663098 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:20.662648 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerDied","Data":"db8ddc07600dc88ccb3ff6207804329f8004f1f86c70c1dc17b64f227bc84169"} Sep 10 09:27:21.669036 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:21.668934 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerStarted","Data":"0b0ef78f8c8b8c2af2551433298d387d5f1b558b653622fad269e76ee3743f68"} Sep 10 09:27:28.700260 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:28.700222 2546 generic.go:356] "Generic (PLEG): container finished" podID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerID="0b0ef78f8c8b8c2af2551433298d387d5f1b558b653622fad269e76ee3743f68" exitCode=0 Sep 10 09:27:28.700712 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:28.700307 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerDied","Data":"0b0ef78f8c8b8c2af2551433298d387d5f1b558b653622fad269e76ee3743f68"} Sep 10 09:27:28.703630 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:28.703607 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 10 09:27:29.713515 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:29.713462 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerStarted","Data":"e3afc236b6c541b1be3c6f6be746d7d36b8fa118f01ab961aed321183dfb27bd"} Sep 10 09:27:29.714041 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:29.713557 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerStarted","Data":"4b89ad545ffb0120dccd627e0a95333cbb8f0b70af45cc308a2dc6bc98828d6b"} Sep 10 09:27:29.714041 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:29.713587 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerStarted","Data":"4c3867618853c4186a830798a1176cc88749a54ab8b2ece6351f5587879858b1"} Sep 10 09:27:29.738352 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:29.738259 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podStartSLOduration=12.738235966 podStartE2EDuration="12.738235966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:27:29.736920624 +0000 UTC m=+788.918609692" watchObservedRunningTime="2026-09-10 09:27:29.738235966 +0000 UTC m=+788.919925033" Sep 10 09:27:30.631313 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:30.631250 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:30.631639 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:30.631613 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:30.631768 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:30.631655 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:30.633441 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:30.633396 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Sep 10 09:27:30.633601 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:30.633448 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 10 09:27:30.633601 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:30.633404 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Sep 10 09:27:33.631437 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:33.631390 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Sep 10 09:27:33.631943 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:33.631458 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 10 09:27:33.631943 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:33.631391 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Sep 10 09:27:36.631757 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:36.631711 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Sep 10 09:27:36.631757 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:36.631737 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 10 09:27:36.632367 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:36.631703 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Sep 10 09:27:38.631256 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:38.631210 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:38.631705 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:38.631319 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:38.631705 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:38.631348 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:39.631533 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:39.631487 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Sep 10 09:27:39.632627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:39.632598 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:39.633427 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:39.633403 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:39.639202 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:39.639167 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:39.768881 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:39.768844 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:42.631636 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:42.631580 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Sep 10 09:27:45.632343 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:45.632256 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:45.633193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:45.633174 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:27:45.924274 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:27:45.924173 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:28:46.882533 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:46.882500 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:28:51.866837 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:51.866798 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9"] Sep 10 09:28:51.867238 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:51.867045 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="registry" containerID="cri-o://4c3867618853c4186a830798a1176cc88749a54ab8b2ece6351f5587879858b1" gracePeriod=30 Sep 10 09:28:51.867238 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:51.867069 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="offline" containerID="cri-o://e3afc236b6c541b1be3c6f6be746d7d36b8fa118f01ab961aed321183dfb27bd" gracePeriod=30 Sep 10 09:28:51.867238 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:51.867093 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" containerID="cri-o://4b89ad545ffb0120dccd627e0a95333cbb8f0b70af45cc308a2dc6bc98828d6b" gracePeriod=30 Sep 10 09:28:51.881457 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:51.881433 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jb5m6"] Sep 10 09:28:51.881596 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:51.881578 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-jb5m6" podUID="745dab2f-3808-4299-bac0-0fb99b44fcdd" containerName="postgres" containerID="cri-o://144e89ad1e29569f39c08743a88a5a027b3c25a1e6cb5fa7a8174357e2c841d3" gracePeriod=30 Sep 10 09:28:51.886048 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:51.886019 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rp5wz"] Sep 10 09:28:51.886246 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:51.886222 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-rp5wz" podUID="283dc883-6a74-4f82-bf45-d5f31b95bed4" containerName="redis" containerID="cri-o://c790e8d2e189e5582bd189fdbde56605993aa19fa805058c0192a58e6a556eff" gracePeriod=30 Sep 10 09:28:51.931510 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:51.931488 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:28:52.065565 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.065488 2546 generic.go:356] "Generic (PLEG): container finished" podID="745dab2f-3808-4299-bac0-0fb99b44fcdd" containerID="144e89ad1e29569f39c08743a88a5a027b3c25a1e6cb5fa7a8174357e2c841d3" exitCode=0 Sep 10 09:28:52.065695 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.065564 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-jb5m6" event={"ID":"745dab2f-3808-4299-bac0-0fb99b44fcdd","Type":"ContainerDied","Data":"144e89ad1e29569f39c08743a88a5a027b3c25a1e6cb5fa7a8174357e2c841d3"} Sep 10 09:28:52.067152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.067130 2546 generic.go:356] "Generic (PLEG): container finished" podID="283dc883-6a74-4f82-bf45-d5f31b95bed4" containerID="c790e8d2e189e5582bd189fdbde56605993aa19fa805058c0192a58e6a556eff" exitCode=0 Sep 10 09:28:52.067269 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.067206 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-rp5wz" event={"ID":"283dc883-6a74-4f82-bf45-d5f31b95bed4","Type":"ContainerDied","Data":"c790e8d2e189e5582bd189fdbde56605993aa19fa805058c0192a58e6a556eff"} Sep 10 09:28:52.138213 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.138178 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:28:52.140709 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.140689 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-rp5wz" Sep 10 09:28:52.234951 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.234911 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tmsh\" (UniqueName: \"kubernetes.io/projected/745dab2f-3808-4299-bac0-0fb99b44fcdd-kube-api-access-5tmsh\") pod \"745dab2f-3808-4299-bac0-0fb99b44fcdd\" (UID: \"745dab2f-3808-4299-bac0-0fb99b44fcdd\") " Sep 10 09:28:52.234951 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.234947 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2bft\" (UniqueName: \"kubernetes.io/projected/283dc883-6a74-4f82-bf45-d5f31b95bed4-kube-api-access-z2bft\") pod \"283dc883-6a74-4f82-bf45-d5f31b95bed4\" (UID: \"283dc883-6a74-4f82-bf45-d5f31b95bed4\") " Sep 10 09:28:52.235299 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.234994 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/745dab2f-3808-4299-bac0-0fb99b44fcdd-postgresdata\") pod \"745dab2f-3808-4299-bac0-0fb99b44fcdd\" (UID: \"745dab2f-3808-4299-bac0-0fb99b44fcdd\") " Sep 10 09:28:52.236977 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.236945 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745dab2f-3808-4299-bac0-0fb99b44fcdd-kube-api-access-5tmsh" (OuterVolumeSpecName: "kube-api-access-5tmsh") pod "745dab2f-3808-4299-bac0-0fb99b44fcdd" (UID: "745dab2f-3808-4299-bac0-0fb99b44fcdd"). InnerVolumeSpecName "kube-api-access-5tmsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:28:52.236977 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.236956 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283dc883-6a74-4f82-bf45-d5f31b95bed4-kube-api-access-z2bft" (OuterVolumeSpecName: "kube-api-access-z2bft") pod "283dc883-6a74-4f82-bf45-d5f31b95bed4" (UID: "283dc883-6a74-4f82-bf45-d5f31b95bed4"). InnerVolumeSpecName "kube-api-access-z2bft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:28:52.264036 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.264007 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745dab2f-3808-4299-bac0-0fb99b44fcdd-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "745dab2f-3808-4299-bac0-0fb99b44fcdd" (UID: "745dab2f-3808-4299-bac0-0fb99b44fcdd"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:28:52.336300 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.336245 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmsh\" (UniqueName: \"kubernetes.io/projected/745dab2f-3808-4299-bac0-0fb99b44fcdd-kube-api-access-5tmsh\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:28:52.336476 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.336309 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2bft\" (UniqueName: \"kubernetes.io/projected/283dc883-6a74-4f82-bf45-d5f31b95bed4-kube-api-access-z2bft\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:28:52.336476 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:52.336326 2546 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/745dab2f-3808-4299-bac0-0fb99b44fcdd-postgresdata\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:28:53.074244 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.074219 2546 generic.go:356] "Generic (PLEG): container finished" podID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerID="4b89ad545ffb0120dccd627e0a95333cbb8f0b70af45cc308a2dc6bc98828d6b" exitCode=0 Sep 10 09:28:53.074561 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.074303 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerDied","Data":"4b89ad545ffb0120dccd627e0a95333cbb8f0b70af45cc308a2dc6bc98828d6b"} Sep 10 09:28:53.075489 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.075463 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-jb5m6" Sep 10 09:28:53.075613 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.075487 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-jb5m6" event={"ID":"745dab2f-3808-4299-bac0-0fb99b44fcdd","Type":"ContainerDied","Data":"a57ea309301dc0f7d5ae41d355cf938623ee0dc88afa5a35381a33c9f213290d"} Sep 10 09:28:53.075613 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.075533 2546 scope.go:117] "RemoveContainer" containerID="144e89ad1e29569f39c08743a88a5a027b3c25a1e6cb5fa7a8174357e2c841d3" Sep 10 09:28:53.076834 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.076758 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-rp5wz" event={"ID":"283dc883-6a74-4f82-bf45-d5f31b95bed4","Type":"ContainerDied","Data":"8740f11a3a50b616854aab45dc4aba94c164a41ecfce5c51fea780454434344c"} Sep 10 09:28:53.076834 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.076794 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-rp5wz" Sep 10 09:28:53.084736 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.084719 2546 scope.go:117] "RemoveContainer" containerID="c790e8d2e189e5582bd189fdbde56605993aa19fa805058c0192a58e6a556eff" Sep 10 09:28:53.095405 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.095385 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jb5m6"] Sep 10 09:28:53.098419 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.098398 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jb5m6"] Sep 10 09:28:53.107897 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.107878 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rp5wz"] Sep 10 09:28:53.111078 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.111058 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-rp5wz"] Sep 10 09:28:53.260748 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.260717 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="283dc883-6a74-4f82-bf45-d5f31b95bed4" path="/var/lib/kubelet/pods/283dc883-6a74-4f82-bf45-d5f31b95bed4/volumes" Sep 10 09:28:53.261252 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:53.261233 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745dab2f-3808-4299-bac0-0fb99b44fcdd" path="/var/lib/kubelet/pods/745dab2f-3808-4299-bac0-0fb99b44fcdd/volumes" Sep 10 09:28:59.765650 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:28:59.765605 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 10 09:29:09.765370 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:09.765324 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 10 09:29:19.765696 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:19.765605 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 10 09:29:19.766069 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:19.765735 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:29:22.198774 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.198704 2546 generic.go:356] "Generic (PLEG): container finished" podID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerID="e3afc236b6c541b1be3c6f6be746d7d36b8fa118f01ab961aed321183dfb27bd" exitCode=137 Sep 10 09:29:22.198774 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.198730 2546 generic.go:356] "Generic (PLEG): container finished" podID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerID="4c3867618853c4186a830798a1176cc88749a54ab8b2ece6351f5587879858b1" exitCode=137 Sep 10 09:29:22.199106 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.198774 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerDied","Data":"e3afc236b6c541b1be3c6f6be746d7d36b8fa118f01ab961aed321183dfb27bd"} Sep 10 09:29:22.199106 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.198824 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerDied","Data":"4c3867618853c4186a830798a1176cc88749a54ab8b2ece6351f5587879858b1"} Sep 10 09:29:22.505454 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.505430 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:29:22.551710 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.551686 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33375e36-23ce-46e0-bd58-1b7f8ed334db-feast-data\") pod \"33375e36-23ce-46e0-bd58-1b7f8ed334db\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " Sep 10 09:29:22.551816 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.551718 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvdt9\" (UniqueName: \"kubernetes.io/projected/33375e36-23ce-46e0-bd58-1b7f8ed334db-kube-api-access-fvdt9\") pod \"33375e36-23ce-46e0-bd58-1b7f8ed334db\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " Sep 10 09:29:22.551816 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.551745 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-offline-tls\") pod \"33375e36-23ce-46e0-bd58-1b7f8ed334db\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " Sep 10 09:29:22.551816 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.551811 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-online-tls\") pod \"33375e36-23ce-46e0-bd58-1b7f8ed334db\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " Sep 10 09:29:22.551976 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.551841 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-registry-tls\") pod \"33375e36-23ce-46e0-bd58-1b7f8ed334db\" (UID: \"33375e36-23ce-46e0-bd58-1b7f8ed334db\") " Sep 10 09:29:22.553828 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.553797 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-online-tls" (OuterVolumeSpecName: "online-tls") pod "33375e36-23ce-46e0-bd58-1b7f8ed334db" (UID: "33375e36-23ce-46e0-bd58-1b7f8ed334db"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:29:22.553926 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.553903 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33375e36-23ce-46e0-bd58-1b7f8ed334db-kube-api-access-fvdt9" (OuterVolumeSpecName: "kube-api-access-fvdt9") pod "33375e36-23ce-46e0-bd58-1b7f8ed334db" (UID: "33375e36-23ce-46e0-bd58-1b7f8ed334db"). InnerVolumeSpecName "kube-api-access-fvdt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:29:22.554175 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.554147 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "33375e36-23ce-46e0-bd58-1b7f8ed334db" (UID: "33375e36-23ce-46e0-bd58-1b7f8ed334db"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:29:22.554175 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.554167 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "33375e36-23ce-46e0-bd58-1b7f8ed334db" (UID: "33375e36-23ce-46e0-bd58-1b7f8ed334db"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:29:22.562607 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.562583 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33375e36-23ce-46e0-bd58-1b7f8ed334db-feast-data" (OuterVolumeSpecName: "feast-data") pod "33375e36-23ce-46e0-bd58-1b7f8ed334db" (UID: "33375e36-23ce-46e0-bd58-1b7f8ed334db"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:29:22.653247 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.653220 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:29:22.653247 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.653243 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:29:22.653406 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.653255 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33375e36-23ce-46e0-bd58-1b7f8ed334db-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:29:22.653406 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.653268 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvdt9\" (UniqueName: \"kubernetes.io/projected/33375e36-23ce-46e0-bd58-1b7f8ed334db-kube-api-access-fvdt9\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:29:22.653406 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:22.653302 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/33375e36-23ce-46e0-bd58-1b7f8ed334db-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:29:23.205193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.205158 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" event={"ID":"33375e36-23ce-46e0-bd58-1b7f8ed334db","Type":"ContainerDied","Data":"e84101e4e5f02839db304b3d67164b486bc596b8d024fbd3e148705646b424ab"} Sep 10 09:29:23.205723 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.205225 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9" Sep 10 09:29:23.205723 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.205230 2546 scope.go:117] "RemoveContainer" containerID="e3afc236b6c541b1be3c6f6be746d7d36b8fa118f01ab961aed321183dfb27bd" Sep 10 09:29:23.213977 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.213953 2546 scope.go:117] "RemoveContainer" containerID="4b89ad545ffb0120dccd627e0a95333cbb8f0b70af45cc308a2dc6bc98828d6b" Sep 10 09:29:23.221685 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.221667 2546 scope.go:117] "RemoveContainer" containerID="4c3867618853c4186a830798a1176cc88749a54ab8b2ece6351f5587879858b1" Sep 10 09:29:23.228706 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.228687 2546 scope.go:117] "RemoveContainer" containerID="0b0ef78f8c8b8c2af2551433298d387d5f1b558b653622fad269e76ee3743f68" Sep 10 09:29:23.235634 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.235612 2546 scope.go:117] "RemoveContainer" containerID="db8ddc07600dc88ccb3ff6207804329f8004f1f86c70c1dc17b64f227bc84169" Sep 10 09:29:23.238928 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.238907 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9"] Sep 10 09:29:23.244679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.244654 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dd4655f-qdvm9"] Sep 10 09:29:23.260589 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:23.260567 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" path="/var/lib/kubelet/pods/33375e36-23ce-46e0-bd58-1b7f8ed334db/volumes" Sep 10 09:29:29.663294 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:29.663243 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:29:29.764523 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:29.764484 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:29:29.826299 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:29.826252 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:29:30.202596 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:29:30.202565 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:30:00.589742 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:30:00.589706 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:22.781678 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.781644 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:22.881584 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.881557 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:22.950317 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950275 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-ds84h"] Sep 10 09:31:22.950573 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950561 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="registry" Sep 10 09:31:22.950627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950574 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="registry" Sep 10 09:31:22.950627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950586 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="feast-apply" Sep 10 09:31:22.950627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950592 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="feast-apply" Sep 10 09:31:22.950627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950599 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" Sep 10 09:31:22.950627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950605 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" Sep 10 09:31:22.950627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950617 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="feast-init" Sep 10 09:31:22.950627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950622 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="feast-init" Sep 10 09:31:22.950627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950628 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="offline" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950633 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="offline" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950644 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283dc883-6a74-4f82-bf45-d5f31b95bed4" containerName="redis" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950650 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="283dc883-6a74-4f82-bf45-d5f31b95bed4" containerName="redis" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950656 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745dab2f-3808-4299-bac0-0fb99b44fcdd" containerName="postgres" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950661 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="745dab2f-3808-4299-bac0-0fb99b44fcdd" containerName="postgres" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950717 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="283dc883-6a74-4f82-bf45-d5f31b95bed4" containerName="redis" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950724 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="online" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950730 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="745dab2f-3808-4299-bac0-0fb99b44fcdd" containerName="postgres" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950738 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="offline" Sep 10 09:31:22.950867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.950744 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="33375e36-23ce-46e0-bd58-1b7f8ed334db" containerName="registry" Sep 10 09:31:22.955135 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.955110 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" Sep 10 09:31:22.958215 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.958190 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-vzqsx\"" Sep 10 09:31:22.958354 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.958253 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 10 09:31:22.958942 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.958924 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 10 09:31:22.968176 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.968157 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-ds84h"] Sep 10 09:31:22.996038 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:22.996014 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:23.002723 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:23.002706 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmllh\" (UniqueName: \"kubernetes.io/projected/b287b376-0374-4e16-bbd6-415a78fa6589-kube-api-access-zmllh\") pod \"redis-69bbd798b6-ds84h\" (UID: \"b287b376-0374-4e16-bbd6-415a78fa6589\") " pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" Sep 10 09:31:23.103847 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:23.103780 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmllh\" (UniqueName: \"kubernetes.io/projected/b287b376-0374-4e16-bbd6-415a78fa6589-kube-api-access-zmllh\") pod \"redis-69bbd798b6-ds84h\" (UID: \"b287b376-0374-4e16-bbd6-415a78fa6589\") " pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" Sep 10 09:31:23.112116 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:23.112090 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmllh\" (UniqueName: \"kubernetes.io/projected/b287b376-0374-4e16-bbd6-415a78fa6589-kube-api-access-zmllh\") pod \"redis-69bbd798b6-ds84h\" (UID: \"b287b376-0374-4e16-bbd6-415a78fa6589\") " pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" Sep 10 09:31:23.270886 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:23.270855 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" Sep 10 09:31:23.538203 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:23.538176 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-ds84h"] Sep 10 09:31:23.540649 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:31:23.540621 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb287b376_0374_4e16_bbd6_415a78fa6589.slice/crio-0b747abb1baf3759f7ccd9246f279dd75daeca84ea395af8ee76fdcaf0ca5d7c WatchSource:0}: Error finding container 0b747abb1baf3759f7ccd9246f279dd75daeca84ea395af8ee76fdcaf0ca5d7c: Status 404 returned error can't find the container with id 0b747abb1baf3759f7ccd9246f279dd75daeca84ea395af8ee76fdcaf0ca5d7c Sep 10 09:31:23.629275 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:23.629249 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" event={"ID":"b287b376-0374-4e16-bbd6-415a78fa6589","Type":"ContainerStarted","Data":"331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5"} Sep 10 09:31:23.629403 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:23.629305 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" event={"ID":"b287b376-0374-4e16-bbd6-415a78fa6589","Type":"ContainerStarted","Data":"0b747abb1baf3759f7ccd9246f279dd75daeca84ea395af8ee76fdcaf0ca5d7c"} Sep 10 09:31:23.643739 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:23.643700 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" podStartSLOduration=1.643684156 podStartE2EDuration="1.643684156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:31:23.642833237 +0000 UTC m=+1022.824522308" watchObservedRunningTime="2026-09-10 09:31:23.643684156 +0000 UTC m=+1022.825373225" Sep 10 09:31:28.352072 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.352043 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kbcv9"] Sep 10 09:31:28.355338 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.355318 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:31:28.357610 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.357581 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 10 09:31:28.363087 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.363065 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kbcv9"] Sep 10 09:31:28.391561 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.391527 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:28.443931 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.443904 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b2c1-dec0-45dd-b074-4cb15968d118-postgresdata\") pod \"postgres-864846d4f5-kbcv9\" (UID: \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\") " pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:31:28.444072 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.443976 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b49b\" (UniqueName: \"kubernetes.io/projected/1cc9b2c1-dec0-45dd-b074-4cb15968d118-kube-api-access-2b49b\") pod \"postgres-864846d4f5-kbcv9\" (UID: \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\") " pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:31:28.545096 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.545052 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b49b\" (UniqueName: \"kubernetes.io/projected/1cc9b2c1-dec0-45dd-b074-4cb15968d118-kube-api-access-2b49b\") pod \"postgres-864846d4f5-kbcv9\" (UID: \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\") " pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:31:28.545271 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.545103 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b2c1-dec0-45dd-b074-4cb15968d118-postgresdata\") pod \"postgres-864846d4f5-kbcv9\" (UID: \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\") " pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:31:28.545527 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.545508 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b2c1-dec0-45dd-b074-4cb15968d118-postgresdata\") pod \"postgres-864846d4f5-kbcv9\" (UID: \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\") " pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:31:28.553826 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.553806 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b49b\" (UniqueName: \"kubernetes.io/projected/1cc9b2c1-dec0-45dd-b074-4cb15968d118-kube-api-access-2b49b\") pod \"postgres-864846d4f5-kbcv9\" (UID: \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\") " pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:31:28.665203 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.665099 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:31:28.931850 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:28.931821 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kbcv9"] Sep 10 09:31:28.934423 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:31:28.934398 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cc9b2c1_dec0_45dd_b074_4cb15968d118.slice/crio-3f06fe4742be1bcf0f71e87315c7c1c60c48684bf0e8ab3357506dace63edfcc WatchSource:0}: Error finding container 3f06fe4742be1bcf0f71e87315c7c1c60c48684bf0e8ab3357506dace63edfcc: Status 404 returned error can't find the container with id 3f06fe4742be1bcf0f71e87315c7c1c60c48684bf0e8ab3357506dace63edfcc Sep 10 09:31:29.654900 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:29.654863 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" event={"ID":"1cc9b2c1-dec0-45dd-b074-4cb15968d118","Type":"ContainerStarted","Data":"ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a"} Sep 10 09:31:29.655264 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:29.654911 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" event={"ID":"1cc9b2c1-dec0-45dd-b074-4cb15968d118","Type":"ContainerStarted","Data":"3f06fe4742be1bcf0f71e87315c7c1c60c48684bf0e8ab3357506dace63edfcc"} Sep 10 09:31:29.670000 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:29.669953 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" podStartSLOduration=1.66993979 podStartE2EDuration="1.66993979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:31:29.669135171 +0000 UTC m=+1028.850824238" watchObservedRunningTime="2026-09-10 09:31:29.66993979 +0000 UTC m=+1028.851628855" Sep 10 09:31:33.795471 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.795421 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:33.872868 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.872835 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:33.880500 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.880470 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf"] Sep 10 09:31:33.883857 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.883840 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:33.886426 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.886402 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 10 09:31:33.886547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.886461 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-mnnjv\"" Sep 10 09:31:33.886547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.886537 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 10 09:31:33.886646 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.886624 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 10 09:31:33.895396 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.895369 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf"] Sep 10 09:31:33.989422 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.989379 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlnxb\" (UniqueName: \"kubernetes.io/projected/0cec77dd-ece2-4b33-afda-d540ae71bff0-kube-api-access-qlnxb\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:33.989422 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.989424 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-registry-tls\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:33.989693 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.989537 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-online-tls\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:33.989693 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:33.989605 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec77dd-ece2-4b33-afda-d540ae71bff0-feast-data\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.090759 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.090660 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-online-tls\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.090759 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.090717 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec77dd-ece2-4b33-afda-d540ae71bff0-feast-data\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.090759 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.090753 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnxb\" (UniqueName: \"kubernetes.io/projected/0cec77dd-ece2-4b33-afda-d540ae71bff0-kube-api-access-qlnxb\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.091048 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.090774 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-registry-tls\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.091048 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:31:34.090825 2546 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 10 09:31:34.091048 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:31:34.090880 2546 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 10 09:31:34.091048 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:31:34.090929 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-online-tls podName:0cec77dd-ece2-4b33-afda-d540ae71bff0 nodeName:}" failed. No retries permitted until 2026-09-10 09:31:34.590907191 +0000 UTC m=+1033.772596234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-online-tls") pod "feast-credit-scoring-545978f49c-j4wxf" (UID: "0cec77dd-ece2-4b33-afda-d540ae71bff0") : secret "feast-credit-scoring-online-tls" not found Sep 10 09:31:34.091048 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:31:34.090950 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-registry-tls podName:0cec77dd-ece2-4b33-afda-d540ae71bff0 nodeName:}" failed. No retries permitted until 2026-09-10 09:31:34.590941333 +0000 UTC m=+1033.772630377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-registry-tls") pod "feast-credit-scoring-545978f49c-j4wxf" (UID: "0cec77dd-ece2-4b33-afda-d540ae71bff0") : secret "feast-credit-scoring-registry-tls" not found Sep 10 09:31:34.091225 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.091169 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec77dd-ece2-4b33-afda-d540ae71bff0-feast-data\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.099944 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.099909 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnxb\" (UniqueName: \"kubernetes.io/projected/0cec77dd-ece2-4b33-afda-d540ae71bff0-kube-api-access-qlnxb\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.234528 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.234492 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:34.335433 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.335393 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:34.595737 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.595698 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-registry-tls\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.595924 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.595775 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-online-tls\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.598085 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.598056 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-online-tls\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.598196 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.598158 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-registry-tls\") pod \"feast-credit-scoring-545978f49c-j4wxf\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:34.793084 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:34.793046 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:35.068332 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:35.068308 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf"] Sep 10 09:31:35.070363 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:31:35.070337 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cec77dd_ece2_4b33_afda_d540ae71bff0.slice/crio-228eb44e99b33c2ef128ead0801dee87a4a669a650a250eb46ed580694575e5e WatchSource:0}: Error finding container 228eb44e99b33c2ef128ead0801dee87a4a669a650a250eb46ed580694575e5e: Status 404 returned error can't find the container with id 228eb44e99b33c2ef128ead0801dee87a4a669a650a250eb46ed580694575e5e Sep 10 09:31:35.680349 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:35.680306 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerStarted","Data":"d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176"} Sep 10 09:31:35.680556 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:35.680359 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerStarted","Data":"228eb44e99b33c2ef128ead0801dee87a4a669a650a250eb46ed580694575e5e"} Sep 10 09:31:37.689408 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:37.689373 2546 generic.go:356] "Generic (PLEG): container finished" podID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerID="d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176" exitCode=0 Sep 10 09:31:37.689781 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:37.689446 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerDied","Data":"d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176"} Sep 10 09:31:38.695119 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:38.695087 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerStarted","Data":"31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e"} Sep 10 09:31:44.720953 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:44.720921 2546 generic.go:356] "Generic (PLEG): container finished" podID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerID="31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e" exitCode=0 Sep 10 09:31:44.721246 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:44.720994 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerDied","Data":"31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e"} Sep 10 09:31:45.728050 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:45.728018 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerStarted","Data":"566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720"} Sep 10 09:31:45.728440 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:45.728073 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerStarted","Data":"b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3"} Sep 10 09:31:45.758447 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:45.758394 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podStartSLOduration=12.758379357 podStartE2EDuration="12.758379357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:31:45.755006198 +0000 UTC m=+1044.936695289" watchObservedRunningTime="2026-09-10 09:31:45.758379357 +0000 UTC m=+1044.940068422" Sep 10 09:31:46.793473 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:46.793436 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:46.793835 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:46.793547 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:46.795019 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:46.794990 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Sep 10 09:31:46.795119 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:46.795048 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Sep 10 09:31:49.793965 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:49.793921 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Sep 10 09:31:49.794373 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:49.793923 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Sep 10 09:31:52.793890 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:52.793843 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Sep 10 09:31:52.799686 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:52.799659 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:52.800053 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:52.800029 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:52.803020 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:52.802991 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:54.794069 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:54.794037 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:55.793815 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:55.793767 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Sep 10 09:31:58.793827 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:58.793793 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:58.794664 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:58.794638 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:31:59.067142 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.067061 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:59.609102 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.609070 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:59.693560 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.693524 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz"] Sep 10 09:31:59.698404 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.698382 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.701414 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.701393 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 10 09:31:59.701566 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.701394 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 10 09:31:59.701715 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.701700 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-p6tmz\"" Sep 10 09:31:59.701760 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.701744 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 10 09:31:59.708275 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.708256 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz"] Sep 10 09:31:59.807355 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.807331 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-feast-data\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.807699 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.807376 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xtk5\" (UniqueName: \"kubernetes.io/projected/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-kube-api-access-6xtk5\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.807699 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.807447 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-registry-tls\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.807699 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.807472 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-online-tls\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.851029 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.850998 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:31:59.908338 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.908267 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-feast-data\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.908441 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.908345 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xtk5\" (UniqueName: \"kubernetes.io/projected/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-kube-api-access-6xtk5\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.908441 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.908398 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-registry-tls\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.908441 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.908425 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-online-tls\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.908618 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:31:59.908522 2546 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Sep 10 09:31:59.908618 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:31:59.908589 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-online-tls podName:83ef9003-c774-4b38-b2ff-f03dcb1dcd96 nodeName:}" failed. No retries permitted until 2026-09-10 09:32:00.408571455 +0000 UTC m=+1059.590260502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-online-tls") pod "feast-driver-ranking-744c69fb78-87cdz" (UID: "83ef9003-c774-4b38-b2ff-f03dcb1dcd96") : secret "feast-driver-ranking-online-tls" not found Sep 10 09:31:59.908739 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.908714 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-feast-data\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.910781 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.910760 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-registry-tls\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.917059 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.917037 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xtk5\" (UniqueName: \"kubernetes.io/projected/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-kube-api-access-6xtk5\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:31:59.933618 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:31:59.933589 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:32:00.265896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:00.265863 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:32:00.411888 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:00.411861 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-online-tls\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:00.414104 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:00.414068 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-online-tls\") pod \"feast-driver-ranking-744c69fb78-87cdz\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:00.608206 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:00.608144 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:00.882366 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:00.882267 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz"] Sep 10 09:32:00.884925 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:32:00.884896 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ef9003_c774_4b38_b2ff_f03dcb1dcd96.slice/crio-969cfd4ff280067b4dbfe08208d9e6d75a6440c8744d94ea64732ae21c12c461 WatchSource:0}: Error finding container 969cfd4ff280067b4dbfe08208d9e6d75a6440c8744d94ea64732ae21c12c461: Status 404 returned error can't find the container with id 969cfd4ff280067b4dbfe08208d9e6d75a6440c8744d94ea64732ae21c12c461 Sep 10 09:32:01.795867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:01.795834 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerStarted","Data":"6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709"} Sep 10 09:32:01.796032 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:01.795881 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerStarted","Data":"969cfd4ff280067b4dbfe08208d9e6d75a6440c8744d94ea64732ae21c12c461"} Sep 10 09:32:05.814057 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:05.814021 2546 generic.go:356] "Generic (PLEG): container finished" podID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerID="6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709" exitCode=0 Sep 10 09:32:05.814493 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:05.814060 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerDied","Data":"6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709"} Sep 10 09:32:06.819836 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:06.819800 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerStarted","Data":"ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744"} Sep 10 09:32:12.844519 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:12.844477 2546 generic.go:356] "Generic (PLEG): container finished" podID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerID="ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744" exitCode=0 Sep 10 09:32:12.844907 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:12.844544 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerDied","Data":"ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744"} Sep 10 09:32:13.852744 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:13.852656 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerStarted","Data":"60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c"} Sep 10 09:32:13.852744 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:13.852728 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerStarted","Data":"74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca"} Sep 10 09:32:13.873624 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:13.873579 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podStartSLOduration=14.873563355 podStartE2EDuration="14.873563355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:32:13.872723001 +0000 UTC m=+1073.054412067" watchObservedRunningTime="2026-09-10 09:32:13.873563355 +0000 UTC m=+1073.055252421" Sep 10 09:32:15.608917 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:15.608865 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:15.609396 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:15.608957 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:15.610533 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:15.610501 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Sep 10 09:32:15.610644 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:15.610549 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Sep 10 09:32:18.608791 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:18.608746 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Sep 10 09:32:18.608791 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:18.608773 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Sep 10 09:32:20.608654 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:20.608617 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:20.608995 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:20.608690 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:21.608392 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:21.608349 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Sep 10 09:32:21.615791 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:21.615764 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:21.619220 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:21.619195 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:24.609307 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:24.609248 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:32:24.610141 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:32:24.610119 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:33:39.270536 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:39.270498 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:41.084935 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.084902 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:41.397401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.397321 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h"] Sep 10 09:33:41.397556 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.397508 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" podUID="ca804916-6f4f-42c0-9c26-ed55673d1315" containerName="manager" containerID="cri-o://c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c" gracePeriod=10 Sep 10 09:33:41.404062 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.404026 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:41.637532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.637498 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:41.659914 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.659853 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" Sep 10 09:33:41.670099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.670067 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf"] Sep 10 09:33:41.670320 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.670271 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="registry" containerID="cri-o://b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3" gracePeriod=30 Sep 10 09:33:41.670442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.670325 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="online" containerID="cri-o://566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720" gracePeriod=30 Sep 10 09:33:41.679542 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.679517 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-758gz\" (UniqueName: \"kubernetes.io/projected/ca804916-6f4f-42c0-9c26-ed55673d1315-kube-api-access-758gz\") pod \"ca804916-6f4f-42c0-9c26-ed55673d1315\" (UID: \"ca804916-6f4f-42c0-9c26-ed55673d1315\") " Sep 10 09:33:41.679692 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.679557 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz"] Sep 10 09:33:41.679831 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.679793 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="registry" containerID="cri-o://74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca" gracePeriod=30 Sep 10 09:33:41.680042 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.679834 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="online" containerID="cri-o://60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c" gracePeriod=30 Sep 10 09:33:41.684240 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.684212 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca804916-6f4f-42c0-9c26-ed55673d1315-kube-api-access-758gz" (OuterVolumeSpecName: "kube-api-access-758gz") pod "ca804916-6f4f-42c0-9c26-ed55673d1315" (UID: "ca804916-6f4f-42c0-9c26-ed55673d1315"). InnerVolumeSpecName "kube-api-access-758gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:33:41.723334 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.723307 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:41.780250 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:41.780226 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-758gz\" (UniqueName: \"kubernetes.io/projected/ca804916-6f4f-42c0-9c26-ed55673d1315-kube-api-access-758gz\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:33:42.230929 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.230886 2546 generic.go:356] "Generic (PLEG): container finished" podID="ca804916-6f4f-42c0-9c26-ed55673d1315" containerID="c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c" exitCode=0 Sep 10 09:33:42.230929 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.230928 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" Sep 10 09:33:42.231479 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.230928 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" event={"ID":"ca804916-6f4f-42c0-9c26-ed55673d1315","Type":"ContainerDied","Data":"c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c"} Sep 10 09:33:42.231479 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.231070 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h" event={"ID":"ca804916-6f4f-42c0-9c26-ed55673d1315","Type":"ContainerDied","Data":"da214728c3f2b835205d20fc96048f5faea999fb23414dc1cd1e4338c568b6f6"} Sep 10 09:33:42.231479 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.231109 2546 scope.go:117] "RemoveContainer" containerID="c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c" Sep 10 09:33:42.239896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.239878 2546 scope.go:117] "RemoveContainer" containerID="c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c" Sep 10 09:33:42.240225 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:33:42.240204 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c\": container with ID starting with c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c not found: ID does not exist" containerID="c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c" Sep 10 09:33:42.240365 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.240232 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c"} err="failed to get container status \"c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c\": rpc error: code = NotFound desc = could not find container \"c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c\": container with ID starting with c5073f4de94adb80d7ccfa7eedb37d4b69e8a4f844ae001cbc404f60a6e9cf2c not found: ID does not exist" Sep 10 09:33:42.253506 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.253475 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h"] Sep 10 09:33:42.259413 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.259384 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-848686b66-z6h4h"] Sep 10 09:33:42.800652 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:42.800568 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Sep 10 09:33:43.238120 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:43.238092 2546 generic.go:356] "Generic (PLEG): container finished" podID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerID="60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c" exitCode=0 Sep 10 09:33:43.238480 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:43.238164 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerDied","Data":"60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c"} Sep 10 09:33:43.240446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:43.240423 2546 generic.go:356] "Generic (PLEG): container finished" podID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerID="566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720" exitCode=0 Sep 10 09:33:43.240547 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:43.240497 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerDied","Data":"566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720"} Sep 10 09:33:43.260494 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:43.260464 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca804916-6f4f-42c0-9c26-ed55673d1315" path="/var/lib/kubelet/pods/ca804916-6f4f-42c0-9c26-ed55673d1315/volumes" Sep 10 09:33:44.216896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.216866 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kbcv9"] Sep 10 09:33:44.217100 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.217027 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" podUID="1cc9b2c1-dec0-45dd-b074-4cb15968d118" containerName="postgres" containerID="cri-o://ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a" gracePeriod=30 Sep 10 09:33:44.221654 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.221630 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-ds84h"] Sep 10 09:33:44.244009 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.243972 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" podUID="b287b376-0374-4e16-bbd6-415a78fa6589" containerName="redis" containerID="cri-o://331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5" gracePeriod=30 Sep 10 09:33:44.265495 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.265256 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:44.461497 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.461472 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:33:44.495013 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.494992 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" Sep 10 09:33:44.501052 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.501029 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b49b\" (UniqueName: \"kubernetes.io/projected/1cc9b2c1-dec0-45dd-b074-4cb15968d118-kube-api-access-2b49b\") pod \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\" (UID: \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\") " Sep 10 09:33:44.501137 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.501091 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b2c1-dec0-45dd-b074-4cb15968d118-postgresdata\") pod \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\" (UID: \"1cc9b2c1-dec0-45dd-b074-4cb15968d118\") " Sep 10 09:33:44.503164 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.503141 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc9b2c1-dec0-45dd-b074-4cb15968d118-kube-api-access-2b49b" (OuterVolumeSpecName: "kube-api-access-2b49b") pod "1cc9b2c1-dec0-45dd-b074-4cb15968d118" (UID: "1cc9b2c1-dec0-45dd-b074-4cb15968d118"). InnerVolumeSpecName "kube-api-access-2b49b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:33:44.532831 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.532800 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc9b2c1-dec0-45dd-b074-4cb15968d118-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "1cc9b2c1-dec0-45dd-b074-4cb15968d118" (UID: "1cc9b2c1-dec0-45dd-b074-4cb15968d118"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:33:44.602357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.602331 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmllh\" (UniqueName: \"kubernetes.io/projected/b287b376-0374-4e16-bbd6-415a78fa6589-kube-api-access-zmllh\") pod \"b287b376-0374-4e16-bbd6-415a78fa6589\" (UID: \"b287b376-0374-4e16-bbd6-415a78fa6589\") " Sep 10 09:33:44.602543 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.602531 2546 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/1cc9b2c1-dec0-45dd-b074-4cb15968d118-postgresdata\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:33:44.602587 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.602544 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b49b\" (UniqueName: \"kubernetes.io/projected/1cc9b2c1-dec0-45dd-b074-4cb15968d118-kube-api-access-2b49b\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:33:44.604157 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.604135 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b287b376-0374-4e16-bbd6-415a78fa6589-kube-api-access-zmllh" (OuterVolumeSpecName: "kube-api-access-zmllh") pod "b287b376-0374-4e16-bbd6-415a78fa6589" (UID: "b287b376-0374-4e16-bbd6-415a78fa6589"). InnerVolumeSpecName "kube-api-access-zmllh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:33:44.702930 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:44.702907 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmllh\" (UniqueName: \"kubernetes.io/projected/b287b376-0374-4e16-bbd6-415a78fa6589-kube-api-access-zmllh\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:33:45.249201 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.249166 2546 generic.go:356] "Generic (PLEG): container finished" podID="1cc9b2c1-dec0-45dd-b074-4cb15968d118" containerID="ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a" exitCode=0 Sep 10 09:33:45.249579 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.249227 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" Sep 10 09:33:45.249579 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.249251 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" event={"ID":"1cc9b2c1-dec0-45dd-b074-4cb15968d118","Type":"ContainerDied","Data":"ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a"} Sep 10 09:33:45.249579 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.249308 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kbcv9" event={"ID":"1cc9b2c1-dec0-45dd-b074-4cb15968d118","Type":"ContainerDied","Data":"3f06fe4742be1bcf0f71e87315c7c1c60c48684bf0e8ab3357506dace63edfcc"} Sep 10 09:33:45.249579 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.249326 2546 scope.go:117] "RemoveContainer" containerID="ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a" Sep 10 09:33:45.250634 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.250598 2546 generic.go:356] "Generic (PLEG): container finished" podID="b287b376-0374-4e16-bbd6-415a78fa6589" containerID="331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5" exitCode=0 Sep 10 09:33:45.250702 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.250637 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" Sep 10 09:33:45.250702 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.250676 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" event={"ID":"b287b376-0374-4e16-bbd6-415a78fa6589","Type":"ContainerDied","Data":"331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5"} Sep 10 09:33:45.250769 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.250722 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-ds84h" event={"ID":"b287b376-0374-4e16-bbd6-415a78fa6589","Type":"ContainerDied","Data":"0b747abb1baf3759f7ccd9246f279dd75daeca84ea395af8ee76fdcaf0ca5d7c"} Sep 10 09:33:45.257414 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.257390 2546 scope.go:117] "RemoveContainer" containerID="ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a" Sep 10 09:33:45.257694 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:33:45.257672 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a\": container with ID starting with ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a not found: ID does not exist" containerID="ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a" Sep 10 09:33:45.257898 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.257703 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a"} err="failed to get container status \"ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a\": rpc error: code = NotFound desc = could not find container \"ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a\": container with ID starting with ab440522e36f81986a8481ade8b84e4a8aa90fc5f5c6e7b6158a8a285fca697a not found: ID does not exist" Sep 10 09:33:45.257898 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.257725 2546 scope.go:117] "RemoveContainer" containerID="331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5" Sep 10 09:33:45.268310 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.268274 2546 scope.go:117] "RemoveContainer" containerID="331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5" Sep 10 09:33:45.268592 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:33:45.268569 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5\": container with ID starting with 331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5 not found: ID does not exist" containerID="331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5" Sep 10 09:33:45.268654 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.268602 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5"} err="failed to get container status \"331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5\": rpc error: code = NotFound desc = could not find container \"331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5\": container with ID starting with 331508ec0bec56bc623190cc775cb06eb6204ac3bf786dcbe827a60fab3277a5 not found: ID does not exist" Sep 10 09:33:45.269024 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.269011 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kbcv9"] Sep 10 09:33:45.273010 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.272991 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kbcv9"] Sep 10 09:33:45.280799 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.280779 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-ds84h"] Sep 10 09:33:45.285221 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:45.285203 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-ds84h"] Sep 10 09:33:46.734065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:46.734027 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:46.794187 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:46.794154 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:47.261627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:47.261592 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc9b2c1-dec0-45dd-b074-4cb15968d118" path="/var/lib/kubelet/pods/1cc9b2c1-dec0-45dd-b074-4cb15968d118/volumes" Sep 10 09:33:47.261918 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:47.261904 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b287b376-0374-4e16-bbd6-415a78fa6589" path="/var/lib/kubelet/pods/b287b376-0374-4e16-bbd6-415a78fa6589/volumes" Sep 10 09:33:51.616812 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:51.616775 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Sep 10 09:33:52.244532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:52.244482 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:52.355942 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:52.355868 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:52.724359 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:52.724260 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:33:52.800712 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:33:52.800678 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Sep 10 09:34:01.616653 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:01.616612 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Sep 10 09:34:02.800524 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:02.800480 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Sep 10 09:34:02.800983 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:02.800622 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:34:06.945111 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:06.945059 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:07.047749 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:07.047720 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:11.617142 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:11.617105 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Sep 10 09:34:11.617519 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:11.617208 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:34:12.329737 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.329715 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:34:12.332360 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.332342 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:34:12.358332 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.358306 2546 generic.go:356] "Generic (PLEG): container finished" podID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerID="74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca" exitCode=137 Sep 10 09:34:12.358461 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.358346 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" Sep 10 09:34:12.358461 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.358380 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerDied","Data":"74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca"} Sep 10 09:34:12.358461 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.358423 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz" event={"ID":"83ef9003-c774-4b38-b2ff-f03dcb1dcd96","Type":"ContainerDied","Data":"969cfd4ff280067b4dbfe08208d9e6d75a6440c8744d94ea64732ae21c12c461"} Sep 10 09:34:12.358461 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.358452 2546 scope.go:117] "RemoveContainer" containerID="60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c" Sep 10 09:34:12.360778 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.360757 2546 generic.go:356] "Generic (PLEG): container finished" podID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerID="b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3" exitCode=137 Sep 10 09:34:12.360888 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.360808 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" Sep 10 09:34:12.360888 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.360841 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerDied","Data":"b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3"} Sep 10 09:34:12.360996 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.360893 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf" event={"ID":"0cec77dd-ece2-4b33-afda-d540ae71bff0","Type":"ContainerDied","Data":"228eb44e99b33c2ef128ead0801dee87a4a669a650a250eb46ed580694575e5e"} Sep 10 09:34:12.368647 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.368630 2546 scope.go:117] "RemoveContainer" containerID="74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca" Sep 10 09:34:12.377928 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.377912 2546 scope.go:117] "RemoveContainer" containerID="ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744" Sep 10 09:34:12.384910 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.384890 2546 scope.go:117] "RemoveContainer" containerID="6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709" Sep 10 09:34:12.391732 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.391718 2546 scope.go:117] "RemoveContainer" containerID="60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c" Sep 10 09:34:12.391974 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:34:12.391956 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c\": container with ID starting with 60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c not found: ID does not exist" containerID="60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c" Sep 10 09:34:12.392040 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.391986 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c"} err="failed to get container status \"60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c\": rpc error: code = NotFound desc = could not find container \"60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c\": container with ID starting with 60b7fd5dbb9a5d4adcab7f71eb9fb1094f41803246ca4f1a35c9b64d21a2c82c not found: ID does not exist" Sep 10 09:34:12.392040 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.392012 2546 scope.go:117] "RemoveContainer" containerID="74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca" Sep 10 09:34:12.392298 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:34:12.392256 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca\": container with ID starting with 74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca not found: ID does not exist" containerID="74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca" Sep 10 09:34:12.392347 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.392304 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca"} err="failed to get container status \"74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca\": rpc error: code = NotFound desc = could not find container \"74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca\": container with ID starting with 74954ff83c882d225e8649f0549e2d5c1fb8074d6659a635e3f16b5cd8d361ca not found: ID does not exist" Sep 10 09:34:12.392347 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.392320 2546 scope.go:117] "RemoveContainer" containerID="ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744" Sep 10 09:34:12.392563 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:34:12.392546 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744\": container with ID starting with ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744 not found: ID does not exist" containerID="ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744" Sep 10 09:34:12.392625 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.392569 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744"} err="failed to get container status \"ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744\": rpc error: code = NotFound desc = could not find container \"ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744\": container with ID starting with ed6bd97db6ef20196cb29efb79a7a0c7837479120cd6c9f3ac7ff223482a1744 not found: ID does not exist" Sep 10 09:34:12.392625 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.392588 2546 scope.go:117] "RemoveContainer" containerID="6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709" Sep 10 09:34:12.392827 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:34:12.392812 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709\": container with ID starting with 6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709 not found: ID does not exist" containerID="6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709" Sep 10 09:34:12.392865 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.392831 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709"} err="failed to get container status \"6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709\": rpc error: code = NotFound desc = could not find container \"6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709\": container with ID starting with 6ee7a45d08da9dffddac4c8a098d8844972a2695af38691dc29589520d46b709 not found: ID does not exist" Sep 10 09:34:12.392865 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.392852 2546 scope.go:117] "RemoveContainer" containerID="566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720" Sep 10 09:34:12.399344 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.399322 2546 scope.go:117] "RemoveContainer" containerID="b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3" Sep 10 09:34:12.402386 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.402367 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-online-tls\") pod \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " Sep 10 09:34:12.402476 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.402413 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlnxb\" (UniqueName: \"kubernetes.io/projected/0cec77dd-ece2-4b33-afda-d540ae71bff0-kube-api-access-qlnxb\") pod \"0cec77dd-ece2-4b33-afda-d540ae71bff0\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " Sep 10 09:34:12.402476 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.402438 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-registry-tls\") pod \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " Sep 10 09:34:12.402588 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.402572 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-registry-tls\") pod \"0cec77dd-ece2-4b33-afda-d540ae71bff0\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " Sep 10 09:34:12.402646 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.402610 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-feast-data\") pod \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " Sep 10 09:34:12.402700 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.402683 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec77dd-ece2-4b33-afda-d540ae71bff0-feast-data\") pod \"0cec77dd-ece2-4b33-afda-d540ae71bff0\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " Sep 10 09:34:12.402763 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.402745 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xtk5\" (UniqueName: \"kubernetes.io/projected/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-kube-api-access-6xtk5\") pod \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\" (UID: \"83ef9003-c774-4b38-b2ff-f03dcb1dcd96\") " Sep 10 09:34:12.402848 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.402830 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-online-tls\") pod \"0cec77dd-ece2-4b33-afda-d540ae71bff0\" (UID: \"0cec77dd-ece2-4b33-afda-d540ae71bff0\") " Sep 10 09:34:12.404568 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.404542 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-feast-data" (OuterVolumeSpecName: "feast-data") pod "83ef9003-c774-4b38-b2ff-f03dcb1dcd96" (UID: "83ef9003-c774-4b38-b2ff-f03dcb1dcd96"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:34:12.404902 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.404877 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "83ef9003-c774-4b38-b2ff-f03dcb1dcd96" (UID: "83ef9003-c774-4b38-b2ff-f03dcb1dcd96"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:34:12.405065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.405052 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cec77dd-ece2-4b33-afda-d540ae71bff0-kube-api-access-qlnxb" (OuterVolumeSpecName: "kube-api-access-qlnxb") pod "0cec77dd-ece2-4b33-afda-d540ae71bff0" (UID: "0cec77dd-ece2-4b33-afda-d540ae71bff0"). InnerVolumeSpecName "kube-api-access-qlnxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:34:12.405413 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.405386 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0cec77dd-ece2-4b33-afda-d540ae71bff0" (UID: "0cec77dd-ece2-4b33-afda-d540ae71bff0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:34:12.405521 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.405472 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-kube-api-access-6xtk5" (OuterVolumeSpecName: "kube-api-access-6xtk5") pod "83ef9003-c774-4b38-b2ff-f03dcb1dcd96" (UID: "83ef9003-c774-4b38-b2ff-f03dcb1dcd96"). InnerVolumeSpecName "kube-api-access-6xtk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:34:12.405680 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.405655 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-online-tls" (OuterVolumeSpecName: "online-tls") pod "83ef9003-c774-4b38-b2ff-f03dcb1dcd96" (UID: "83ef9003-c774-4b38-b2ff-f03dcb1dcd96"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:34:12.406202 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.406179 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-online-tls" (OuterVolumeSpecName: "online-tls") pod "0cec77dd-ece2-4b33-afda-d540ae71bff0" (UID: "0cec77dd-ece2-4b33-afda-d540ae71bff0"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:34:12.407760 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.407742 2546 scope.go:117] "RemoveContainer" containerID="31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e" Sep 10 09:34:12.412427 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.412400 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cec77dd-ece2-4b33-afda-d540ae71bff0-feast-data" (OuterVolumeSpecName: "feast-data") pod "0cec77dd-ece2-4b33-afda-d540ae71bff0" (UID: "0cec77dd-ece2-4b33-afda-d540ae71bff0"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:34:12.414169 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.414152 2546 scope.go:117] "RemoveContainer" containerID="d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176" Sep 10 09:34:12.420443 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.420417 2546 scope.go:117] "RemoveContainer" containerID="566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720" Sep 10 09:34:12.420674 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:34:12.420653 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720\": container with ID starting with 566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720 not found: ID does not exist" containerID="566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720" Sep 10 09:34:12.420723 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.420680 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720"} err="failed to get container status \"566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720\": rpc error: code = NotFound desc = could not find container \"566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720\": container with ID starting with 566859499dc11234dfe70569918b6cd9e9fbf7e484ff318a8cfcdf6695f51720 not found: ID does not exist" Sep 10 09:34:12.420723 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.420695 2546 scope.go:117] "RemoveContainer" containerID="b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3" Sep 10 09:34:12.420925 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:34:12.420910 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3\": container with ID starting with b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3 not found: ID does not exist" containerID="b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3" Sep 10 09:34:12.420979 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.420929 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3"} err="failed to get container status \"b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3\": rpc error: code = NotFound desc = could not find container \"b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3\": container with ID starting with b0a56ea6151024b80a667deb51918a15376b1ebacd6f8e205ded59ef4d1347d3 not found: ID does not exist" Sep 10 09:34:12.420979 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.420944 2546 scope.go:117] "RemoveContainer" containerID="31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e" Sep 10 09:34:12.421168 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:34:12.421152 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e\": container with ID starting with 31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e not found: ID does not exist" containerID="31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e" Sep 10 09:34:12.421233 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.421177 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e"} err="failed to get container status \"31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e\": rpc error: code = NotFound desc = could not find container \"31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e\": container with ID starting with 31a3f81bfd601f169867a1309de6775d559be05d48707b931611d6e4db85441e not found: ID does not exist" Sep 10 09:34:12.421233 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.421197 2546 scope.go:117] "RemoveContainer" containerID="d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176" Sep 10 09:34:12.421460 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:34:12.421443 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176\": container with ID starting with d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176 not found: ID does not exist" containerID="d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176" Sep 10 09:34:12.421504 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.421466 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176"} err="failed to get container status \"d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176\": rpc error: code = NotFound desc = could not find container \"d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176\": container with ID starting with d7b4ad6f4d93d495158ecbf07fd13b3f83483ac7bb7f6f877294ec304bbac176 not found: ID does not exist" Sep 10 09:34:12.504016 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.503983 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:34:12.504016 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.504013 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:34:12.504193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.504032 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlnxb\" (UniqueName: \"kubernetes.io/projected/0cec77dd-ece2-4b33-afda-d540ae71bff0-kube-api-access-qlnxb\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:34:12.504193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.504041 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:34:12.504193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.504050 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0cec77dd-ece2-4b33-afda-d540ae71bff0-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:34:12.504193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.504057 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:34:12.504193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.504066 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0cec77dd-ece2-4b33-afda-d540ae71bff0-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:34:12.504193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.504076 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xtk5\" (UniqueName: \"kubernetes.io/projected/83ef9003-c774-4b38-b2ff-f03dcb1dcd96-kube-api-access-6xtk5\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:34:12.679961 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.679926 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz"] Sep 10 09:34:12.683556 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.683531 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-744c69fb78-87cdz"] Sep 10 09:34:12.692818 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.692793 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf"] Sep 10 09:34:12.695089 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:12.695063 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-545978f49c-j4wxf"] Sep 10 09:34:13.260518 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:13.260490 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" path="/var/lib/kubelet/pods/0cec77dd-ece2-4b33-afda-d540ae71bff0/volumes" Sep 10 09:34:13.261026 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:13.261012 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" path="/var/lib/kubelet/pods/83ef9003-c774-4b38-b2ff-f03dcb1dcd96/volumes" Sep 10 09:34:18.274591 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.274553 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:18.373989 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.373956 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:18.400899 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.400872 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp"] Sep 10 09:34:18.401155 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401143 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="feast-init" Sep 10 09:34:18.401200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401157 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="feast-init" Sep 10 09:34:18.401200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401165 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="feast-apply" Sep 10 09:34:18.401200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401171 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="feast-apply" Sep 10 09:34:18.401200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401182 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b287b376-0374-4e16-bbd6-415a78fa6589" containerName="redis" Sep 10 09:34:18.401200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401191 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b287b376-0374-4e16-bbd6-415a78fa6589" containerName="redis" Sep 10 09:34:18.401200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401202 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca804916-6f4f-42c0-9c26-ed55673d1315" containerName="manager" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401207 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca804916-6f4f-42c0-9c26-ed55673d1315" containerName="manager" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401214 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="feast-apply" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401219 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="feast-apply" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401228 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="registry" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401233 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="registry" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401242 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="feast-init" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401248 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="feast-init" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401253 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="online" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401261 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="online" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401273 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="online" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401288 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="online" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401299 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="registry" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401303 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="registry" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401312 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc9b2c1-dec0-45dd-b074-4cb15968d118" containerName="postgres" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401318 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc9b2c1-dec0-45dd-b074-4cb15968d118" containerName="postgres" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401370 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="registry" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401379 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cec77dd-ece2-4b33-afda-d540ae71bff0" containerName="online" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401385 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b287b376-0374-4e16-bbd6-415a78fa6589" containerName="redis" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401392 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="online" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401398 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc9b2c1-dec0-45dd-b074-4cb15968d118" containerName="postgres" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401404 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ef9003-c774-4b38-b2ff-f03dcb1dcd96" containerName="registry" Sep 10 09:34:18.401410 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.401408 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca804916-6f4f-42c0-9c26-ed55673d1315" containerName="manager" Sep 10 09:34:18.406855 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.406835 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.409502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.409349 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 10 09:34:18.409502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.409365 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 10 09:34:18.409502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.409378 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 10 09:34:18.409502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.409387 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 10 09:34:18.409502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.409394 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 10 09:34:18.409502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.409406 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-987sz\"" Sep 10 09:34:18.409502 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.409348 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 10 09:34:18.416045 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.416018 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp"] Sep 10 09:34:18.504760 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.504734 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:18.548567 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.548501 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5216e14c-dcfb-49cb-85a3-cf828762ebeb-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.548567 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.548537 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.548567 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.548563 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.548789 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.548597 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz9gj\" (UniqueName: \"kubernetes.io/projected/5216e14c-dcfb-49cb-85a3-cf828762ebeb-kube-api-access-lz9gj\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.548789 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.548668 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.548789 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.548742 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.604374 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.604346 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:18.649791 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.649769 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5216e14c-dcfb-49cb-85a3-cf828762ebeb-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.649895 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.649801 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.649895 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.649817 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.650008 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.649932 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz9gj\" (UniqueName: \"kubernetes.io/projected/5216e14c-dcfb-49cb-85a3-cf828762ebeb-kube-api-access-lz9gj\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.650008 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.649988 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.650125 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.650073 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.650215 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.650198 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5216e14c-dcfb-49cb-85a3-cf828762ebeb-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.652231 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.652209 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.652375 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.652273 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.652434 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.652391 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.652556 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.652535 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.660260 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.660240 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz9gj\" (UniqueName: \"kubernetes.io/projected/5216e14c-dcfb-49cb-85a3-cf828762ebeb-kube-api-access-lz9gj\") pod \"feast-simple-feast-setup-57576b8fcf-n6hdp\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.704297 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.704256 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:18.722235 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.722212 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:18.997480 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:18.997449 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp"] Sep 10 09:34:19.001161 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:34:19.001135 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5216e14c_dcfb_49cb_85a3_cf828762ebeb.slice/crio-6d7dd3dfcf609221f47a45098b3faaf0825d025635cb9aeefc51dda3aaf697dc WatchSource:0}: Error finding container 6d7dd3dfcf609221f47a45098b3faaf0825d025635cb9aeefc51dda3aaf697dc: Status 404 returned error can't find the container with id 6d7dd3dfcf609221f47a45098b3faaf0825d025635cb9aeefc51dda3aaf697dc Sep 10 09:34:19.002996 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:19.002979 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 09:34:19.389787 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:19.389721 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerStarted","Data":"6d7dd3dfcf609221f47a45098b3faaf0825d025635cb9aeefc51dda3aaf697dc"} Sep 10 09:34:20.439918 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:20.439860 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:21.928891 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:21.928795 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:21.988758 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:21.988723 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:22.222603 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:22.222511 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:34:32.449758 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:32.449665 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerStarted","Data":"ca767690d69afe8695902d0872d7dd09495e4edd2565941756158a30010ea8f4"} Sep 10 09:34:37.472322 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:37.472269 2546 generic.go:356] "Generic (PLEG): container finished" podID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerID="ca767690d69afe8695902d0872d7dd09495e4edd2565941756158a30010ea8f4" exitCode=0 Sep 10 09:34:37.472322 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:37.472325 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerDied","Data":"ca767690d69afe8695902d0872d7dd09495e4edd2565941756158a30010ea8f4"} Sep 10 09:34:38.478748 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:38.478709 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerStarted","Data":"6bb6d803908c5b9fab2068efa23adef4535cbee048fcc1deef9630c72998a9e6"} Sep 10 09:34:43.499513 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:43.499485 2546 generic.go:356] "Generic (PLEG): container finished" podID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerID="6bb6d803908c5b9fab2068efa23adef4535cbee048fcc1deef9630c72998a9e6" exitCode=0 Sep 10 09:34:43.499834 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:43.499558 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerDied","Data":"6bb6d803908c5b9fab2068efa23adef4535cbee048fcc1deef9630c72998a9e6"} Sep 10 09:34:44.509657 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:44.509616 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerStarted","Data":"741209b578c82dafec4df13a07d30e7d10c16aac5dc525f17078bbc0ac4f9523"} Sep 10 09:34:44.510166 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:44.509681 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerStarted","Data":"b8d7f8c9d980c7776048d7bb3cb1b7cde8d0a2da31a32f05424840b46ee3027c"} Sep 10 09:34:44.510166 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:44.509707 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerStarted","Data":"59b5c2115937d1b715a18be60951f557f9b52601a0515aae2708005a6da5da98"} Sep 10 09:34:44.510166 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:44.509726 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerStarted","Data":"8c287f9a50feb805db23e4094c8c2c98888cb7d0cf2764c6ac222a7b0c8a6acb"} Sep 10 09:34:44.535953 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:44.535876 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podStartSLOduration=13.442923695 podStartE2EDuration="26.535853147s" totalImagesPullingTime="13.092929452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:34:44.5342331 +0000 UTC m=+1223.715922178" watchObservedRunningTime="2026-09-10 09:34:44.535853147 +0000 UTC m=+1223.717542235" Sep 10 09:34:45.722595 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:45.722549 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:45.723065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:45.722609 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:45.723065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:45.722631 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:45.723183 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:45.723126 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:45.724903 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:45.724869 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Sep 10 09:34:45.725049 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:45.724908 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 10 09:34:45.725049 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:45.724971 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Sep 10 09:34:45.725231 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:45.725198 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused" Sep 10 09:34:48.722748 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:48.722706 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:48.723685 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:48.723254 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 10 09:34:48.723685 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:48.723299 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:48.723685 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:48.723072 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused" Sep 10 09:34:48.723685 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:48.722973 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Sep 10 09:34:48.723685 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:48.722994 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Sep 10 09:34:48.724063 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:48.723721 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:48.724063 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:48.723750 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:51.723841 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:51.723473 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Sep 10 09:34:51.724333 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:51.724074 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:51.724585 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:51.724558 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:51.724973 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:51.724952 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:51.730166 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:51.730140 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:52.550065 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:52.550035 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:52.552767 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:52.552738 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:54.723446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:54.723417 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:34:55.560705 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:34:55.560672 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:35:26.418425 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:26.418333 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:35:28.101191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:28.101156 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:35:57.687811 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:57.687778 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp"] Sep 10 09:35:57.688408 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:57.688052 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="registry" containerID="cri-o://8c287f9a50feb805db23e4094c8c2c98888cb7d0cf2764c6ac222a7b0c8a6acb" gracePeriod=30 Sep 10 09:35:57.688408 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:57.688192 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="offline" containerID="cri-o://b8d7f8c9d980c7776048d7bb3cb1b7cde8d0a2da31a32f05424840b46ee3027c" gracePeriod=30 Sep 10 09:35:57.688408 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:57.688203 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="online" containerID="cri-o://59b5c2115937d1b715a18be60951f557f9b52601a0515aae2708005a6da5da98" gracePeriod=30 Sep 10 09:35:57.688408 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:57.688205 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" containerID="cri-o://741209b578c82dafec4df13a07d30e7d10c16aac5dc525f17078bbc0ac4f9523" gracePeriod=30 Sep 10 09:35:57.692671 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:57.692645 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:35:57.752149 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:57.752114 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:35:57.967715 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:57.967633 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:35:58.066719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.066668 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:35:58.083675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.083641 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6"] Sep 10 09:35:58.088155 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.088130 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.090520 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.090499 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bcnkw\"" Sep 10 09:35:58.104369 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.101841 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6"] Sep 10 09:35:58.190872 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.190833 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxc4\" (UniqueName: \"kubernetes.io/projected/9a8ad248-5f07-428d-aa29-16f330c2f449-kube-api-access-znxc4\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.190872 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.190877 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8ad248-5f07-428d-aa29-16f330c2f449-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.191128 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.190923 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.191128 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.190959 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.191128 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.190982 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.191128 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.191076 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.266267 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.266233 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:35:58.292056 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.292021 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.292252 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.292081 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.292252 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.292127 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxc4\" (UniqueName: \"kubernetes.io/projected/9a8ad248-5f07-428d-aa29-16f330c2f449-kube-api-access-znxc4\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.292252 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.292156 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8ad248-5f07-428d-aa29-16f330c2f449-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.292252 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.292192 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.292252 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.292241 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.292976 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.292947 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8ad248-5f07-428d-aa29-16f330c2f449-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.295000 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.294952 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.295171 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.295148 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.295388 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.295371 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.295665 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.295642 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.300559 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.300535 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxc4\" (UniqueName: \"kubernetes.io/projected/9a8ad248-5f07-428d-aa29-16f330c2f449-kube-api-access-znxc4\") pod \"feast-simple-feast-setup-57576b8fcf-qcbr6\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.361426 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.361393 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:35:58.397422 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.397388 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:35:58.674084 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:35:58.674050 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a8ad248_5f07_428d_aa29_16f330c2f449.slice/crio-4ebbcdb0eb875513990515de067ac7e8d1d601b73d22e6956a3b47b1bc182dc7 WatchSource:0}: Error finding container 4ebbcdb0eb875513990515de067ac7e8d1d601b73d22e6956a3b47b1bc182dc7: Status 404 returned error can't find the container with id 4ebbcdb0eb875513990515de067ac7e8d1d601b73d22e6956a3b47b1bc182dc7 Sep 10 09:35:58.674211 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.674181 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6"] Sep 10 09:35:58.796691 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.796618 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerStarted","Data":"2d5726d3258da6850d57f6df2a3c99998fd4b57bb17f2d7aaec42567b646c735"} Sep 10 09:35:58.796691 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.796670 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerStarted","Data":"4ebbcdb0eb875513990515de067ac7e8d1d601b73d22e6956a3b47b1bc182dc7"} Sep 10 09:35:58.800071 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.800045 2546 generic.go:356] "Generic (PLEG): container finished" podID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerID="741209b578c82dafec4df13a07d30e7d10c16aac5dc525f17078bbc0ac4f9523" exitCode=0 Sep 10 09:35:58.800210 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.800070 2546 generic.go:356] "Generic (PLEG): container finished" podID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerID="59b5c2115937d1b715a18be60951f557f9b52601a0515aae2708005a6da5da98" exitCode=0 Sep 10 09:35:58.800210 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.800119 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerDied","Data":"741209b578c82dafec4df13a07d30e7d10c16aac5dc525f17078bbc0ac4f9523"} Sep 10 09:35:58.800210 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:35:58.800161 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerDied","Data":"59b5c2115937d1b715a18be60951f557f9b52601a0515aae2708005a6da5da98"} Sep 10 09:36:02.550005 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:02.549964 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 10 09:36:03.824251 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:03.824215 2546 generic.go:356] "Generic (PLEG): container finished" podID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerID="2d5726d3258da6850d57f6df2a3c99998fd4b57bb17f2d7aaec42567b646c735" exitCode=0 Sep 10 09:36:03.824630 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:03.824312 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerDied","Data":"2d5726d3258da6850d57f6df2a3c99998fd4b57bb17f2d7aaec42567b646c735"} Sep 10 09:36:04.830984 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:04.830952 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerStarted","Data":"5b2659207a11afac6f1ddafa2ee7b05c9313bfe2caeee1902b62e52efcdaf5c1"} Sep 10 09:36:05.560626 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:05.560583 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Sep 10 09:36:09.859627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:09.859549 2546 generic.go:356] "Generic (PLEG): container finished" podID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerID="5b2659207a11afac6f1ddafa2ee7b05c9313bfe2caeee1902b62e52efcdaf5c1" exitCode=0 Sep 10 09:36:09.859627 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:09.859602 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerDied","Data":"5b2659207a11afac6f1ddafa2ee7b05c9313bfe2caeee1902b62e52efcdaf5c1"} Sep 10 09:36:10.870487 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:10.870434 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerStarted","Data":"a21c5543cd93e0d6918c5ce065a1fd0bcbfcdfd0c8a2e619221fcd288769daea"} Sep 10 09:36:10.870974 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:10.870496 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerStarted","Data":"a2f990ba522745c98a0cd8cced3c00ccb34622e9fe20aef06b47d55caed27c0f"} Sep 10 09:36:10.870974 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:10.870515 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerStarted","Data":"006205b98adc481a822f10642e2c8be2e479e0b818073e812442c2734171dec3"} Sep 10 09:36:10.870974 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:10.870531 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerStarted","Data":"a3dc9efa8e204fad167ade9a47acc54a30ffcfd741f42ad1fefa7e4573e4057e"} Sep 10 09:36:10.894911 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:10.894844 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podStartSLOduration=12.894821061 podStartE2EDuration="12.894821061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:36:10.894690382 +0000 UTC m=+1310.076379449" watchObservedRunningTime="2026-09-10 09:36:10.894821061 +0000 UTC m=+1310.076510127" Sep 10 09:36:12.550331 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:12.550260 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 10 09:36:13.398768 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:13.398731 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:13.398963 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:13.398804 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:13.398963 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:13.398825 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:13.398963 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:13.398842 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:13.400415 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:13.400376 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Sep 10 09:36:13.400415 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:13.400397 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Sep 10 09:36:13.400613 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:13.400404 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused" Sep 10 09:36:13.400613 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:13.400459 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 10 09:36:15.561173 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:15.561114 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Sep 10 09:36:16.398302 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:16.398244 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Sep 10 09:36:16.398590 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:16.398251 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused" Sep 10 09:36:16.398715 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:16.398291 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Sep 10 09:36:16.398715 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:16.398478 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 10 09:36:16.909016 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:16.908959 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:36:18.397750 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:18.397708 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:18.398097 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:18.397766 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:18.398097 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:18.397780 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:18.398097 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:18.397792 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:19.398631 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:19.398592 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:19.399168 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:19.399140 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:19.399294 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:19.399267 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:19.399519 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:19.399502 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:19.403946 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:19.403928 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:19.915829 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:19.915802 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:19.915968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:19.915951 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:19.918743 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:19.918730 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:36:22.549890 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:22.549850 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 10 09:36:22.550359 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:22.549975 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:36:25.560366 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:25.560326 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused" Sep 10 09:36:25.560739 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:25.560428 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:36:27.946824 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:27.946762 2546 generic.go:356] "Generic (PLEG): container finished" podID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerID="b8d7f8c9d980c7776048d7bb3cb1b7cde8d0a2da31a32f05424840b46ee3027c" exitCode=137 Sep 10 09:36:27.946824 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:27.946785 2546 generic.go:356] "Generic (PLEG): container finished" podID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerID="8c287f9a50feb805db23e4094c8c2c98888cb7d0cf2764c6ac222a7b0c8a6acb" exitCode=137 Sep 10 09:36:27.947128 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:27.946831 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerDied","Data":"b8d7f8c9d980c7776048d7bb3cb1b7cde8d0a2da31a32f05424840b46ee3027c"} Sep 10 09:36:27.947128 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:27.946869 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerDied","Data":"8c287f9a50feb805db23e4094c8c2c98888cb7d0cf2764c6ac222a7b0c8a6acb"} Sep 10 09:36:28.324712 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.324688 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:36:28.421238 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.421213 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-offline-tls\") pod \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " Sep 10 09:36:28.421431 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.421274 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-registry-tls\") pod \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " Sep 10 09:36:28.421431 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.421334 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-online-tls\") pod \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " Sep 10 09:36:28.421431 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.421364 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5216e14c-dcfb-49cb-85a3-cf828762ebeb-feast-data\") pod \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " Sep 10 09:36:28.421431 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.421386 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-ui-tls\") pod \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " Sep 10 09:36:28.421431 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.421421 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz9gj\" (UniqueName: \"kubernetes.io/projected/5216e14c-dcfb-49cb-85a3-cf828762ebeb-kube-api-access-lz9gj\") pod \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\" (UID: \"5216e14c-dcfb-49cb-85a3-cf828762ebeb\") " Sep 10 09:36:28.421903 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.421873 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5216e14c-dcfb-49cb-85a3-cf828762ebeb-feast-data" (OuterVolumeSpecName: "feast-data") pod "5216e14c-dcfb-49cb-85a3-cf828762ebeb" (UID: "5216e14c-dcfb-49cb-85a3-cf828762ebeb"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:36:28.423550 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.423523 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5216e14c-dcfb-49cb-85a3-cf828762ebeb" (UID: "5216e14c-dcfb-49cb-85a3-cf828762ebeb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:36:28.423650 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.423558 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5216e14c-dcfb-49cb-85a3-cf828762ebeb" (UID: "5216e14c-dcfb-49cb-85a3-cf828762ebeb"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:36:28.423650 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.423605 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5216e14c-dcfb-49cb-85a3-cf828762ebeb" (UID: "5216e14c-dcfb-49cb-85a3-cf828762ebeb"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:36:28.423814 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.423794 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-online-tls" (OuterVolumeSpecName: "online-tls") pod "5216e14c-dcfb-49cb-85a3-cf828762ebeb" (UID: "5216e14c-dcfb-49cb-85a3-cf828762ebeb"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:36:28.423915 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.423898 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5216e14c-dcfb-49cb-85a3-cf828762ebeb-kube-api-access-lz9gj" (OuterVolumeSpecName: "kube-api-access-lz9gj") pod "5216e14c-dcfb-49cb-85a3-cf828762ebeb" (UID: "5216e14c-dcfb-49cb-85a3-cf828762ebeb"). InnerVolumeSpecName "kube-api-access-lz9gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:36:28.521931 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.521909 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:36:28.521931 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.521931 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:36:28.522149 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.521939 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5216e14c-dcfb-49cb-85a3-cf828762ebeb-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:36:28.522149 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.521946 2546 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-ui-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:36:28.522149 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.521954 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz9gj\" (UniqueName: \"kubernetes.io/projected/5216e14c-dcfb-49cb-85a3-cf828762ebeb-kube-api-access-lz9gj\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:36:28.522149 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.521965 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5216e14c-dcfb-49cb-85a3-cf828762ebeb-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:36:28.953805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.953743 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" event={"ID":"5216e14c-dcfb-49cb-85a3-cf828762ebeb","Type":"ContainerDied","Data":"6d7dd3dfcf609221f47a45098b3faaf0825d025635cb9aeefc51dda3aaf697dc"} Sep 10 09:36:28.953805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.953798 2546 scope.go:117] "RemoveContainer" containerID="741209b578c82dafec4df13a07d30e7d10c16aac5dc525f17078bbc0ac4f9523" Sep 10 09:36:28.954191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.953759 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp" Sep 10 09:36:28.962577 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.962426 2546 scope.go:117] "RemoveContainer" containerID="b8d7f8c9d980c7776048d7bb3cb1b7cde8d0a2da31a32f05424840b46ee3027c" Sep 10 09:36:28.969677 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.969661 2546 scope.go:117] "RemoveContainer" containerID="59b5c2115937d1b715a18be60951f557f9b52601a0515aae2708005a6da5da98" Sep 10 09:36:28.975802 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.975780 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp"] Sep 10 09:36:28.976683 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.976670 2546 scope.go:117] "RemoveContainer" containerID="8c287f9a50feb805db23e4094c8c2c98888cb7d0cf2764c6ac222a7b0c8a6acb" Sep 10 09:36:28.981096 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.981071 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-n6hdp"] Sep 10 09:36:28.983708 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.983694 2546 scope.go:117] "RemoveContainer" containerID="6bb6d803908c5b9fab2068efa23adef4535cbee048fcc1deef9630c72998a9e6" Sep 10 09:36:28.990234 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:28.990219 2546 scope.go:117] "RemoveContainer" containerID="ca767690d69afe8695902d0872d7dd09495e4edd2565941756158a30010ea8f4" Sep 10 09:36:29.260269 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:36:29.260245 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" path="/var/lib/kubelet/pods/5216e14c-dcfb-49cb-85a3-cf828762ebeb/volumes" Sep 10 09:37:22.561679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:22.561640 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:37:22.617126 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:22.617098 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:37:22.924458 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:22.924365 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:37:23.030346 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.030319 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:37:23.109044 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.109015 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:37:23.116379 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116354 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx"] Sep 10 09:37:23.116676 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116662 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="registry" Sep 10 09:37:23.116719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116678 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="registry" Sep 10 09:37:23.116719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116690 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" Sep 10 09:37:23.116719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116696 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" Sep 10 09:37:23.116719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116707 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="feast-apply" Sep 10 09:37:23.116719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116713 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="feast-apply" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116722 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="feast-init" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116729 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="feast-init" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116736 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="offline" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116744 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="offline" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116753 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="online" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116758 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="online" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116823 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="ui" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116834 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="offline" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116841 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="registry" Sep 10 09:37:23.116877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.116846 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5216e14c-dcfb-49cb-85a3-cf828762ebeb" containerName="online" Sep 10 09:37:23.120846 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.120832 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.122800 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.122778 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 10 09:37:23.122900 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.122824 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 10 09:37:23.122900 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.122889 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 10 09:37:23.123222 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.123203 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-6cfdw\"" Sep 10 09:37:23.123519 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.123503 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 10 09:37:23.123606 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.123524 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 10 09:37:23.123745 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.123728 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 10 09:37:23.131094 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.131073 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx"] Sep 10 09:37:23.189086 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.189037 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.189086 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.189067 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.189198 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.189089 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.189198 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.189110 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmmv\" (UniqueName: \"kubernetes.io/projected/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-kube-api-access-qbmmv\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.189268 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.189193 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.189268 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.189224 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.217182 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.217163 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:37:23.289931 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.289911 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.290015 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.289940 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.290015 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.289964 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmmv\" (UniqueName: \"kubernetes.io/projected/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-kube-api-access-qbmmv\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.290099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.290013 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.290099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.290041 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.290099 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:37:23.290055 2546 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 10 09:37:23.290099 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.290087 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.290318 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:37:23.290131 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-online-tls podName:8ed0a908-6413-4795-99a8-7d6aa4a6b12a nodeName:}" failed. No retries permitted until 2026-09-10 09:37:23.790112837 +0000 UTC m=+1382.971801943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-online-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" (UID: "8ed0a908-6413-4795-99a8-7d6aa4a6b12a") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 10 09:37:23.290458 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.290439 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.290640 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.290617 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.292458 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.292441 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.292592 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.292576 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.297517 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.297496 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmmv\" (UniqueName: \"kubernetes.io/projected/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-kube-api-access-qbmmv\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.342014 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.341990 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:37:23.792087 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.792064 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:23.794179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:23.794161 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-6wdxx\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:24.029623 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:24.029595 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:24.304791 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:24.304768 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx"] Sep 10 09:37:24.307034 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:37:24.307005 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ed0a908_6413_4795_99a8_7d6aa4a6b12a.slice/crio-9a7398f19ce4f57a84495b5c2eaaefebceda619cdb2e199cc6c4f5bc44e72673 WatchSource:0}: Error finding container 9a7398f19ce4f57a84495b5c2eaaefebceda619cdb2e199cc6c4f5bc44e72673: Status 404 returned error can't find the container with id 9a7398f19ce4f57a84495b5c2eaaefebceda619cdb2e199cc6c4f5bc44e72673 Sep 10 09:37:25.177663 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:25.177625 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerStarted","Data":"fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29"} Sep 10 09:37:25.178027 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:25.177671 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerStarted","Data":"9a7398f19ce4f57a84495b5c2eaaefebceda619cdb2e199cc6c4f5bc44e72673"} Sep 10 09:37:29.199105 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:29.199073 2546 generic.go:356] "Generic (PLEG): container finished" podID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerID="fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29" exitCode=0 Sep 10 09:37:29.199447 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:29.199156 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerDied","Data":"fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29"} Sep 10 09:37:30.205147 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:30.205111 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerStarted","Data":"f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33"} Sep 10 09:37:50.278216 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:50.278133 2546 generic.go:356] "Generic (PLEG): container finished" podID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerID="f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33" exitCode=0 Sep 10 09:37:50.278216 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:50.278197 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerDied","Data":"f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33"} Sep 10 09:37:51.286154 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:51.286113 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerStarted","Data":"049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe"} Sep 10 09:37:51.286641 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:51.286172 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerStarted","Data":"f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16"} Sep 10 09:37:51.286641 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:51.286186 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerStarted","Data":"a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8"} Sep 10 09:37:51.308218 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:51.308157 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podStartSLOduration=28.308136974 podStartE2EDuration="28.308136974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:37:51.306366845 +0000 UTC m=+1410.488055911" watchObservedRunningTime="2026-09-10 09:37:51.308136974 +0000 UTC m=+1410.489826039" Sep 10 09:37:54.030400 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:54.030369 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:54.031059 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:54.031030 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:54.031059 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:54.031061 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:54.031244 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:54.031077 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:54.031244 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:54.031094 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:54.031244 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:54.031111 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:37:54.032015 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:54.031980 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused" Sep 10 09:37:54.032015 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:54.032006 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 10 09:37:54.032167 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:54.031992 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 10 09:37:57.030788 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:57.030737 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused" Sep 10 09:37:57.031310 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:57.030749 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 10 09:37:57.031310 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:37:57.030816 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 10 09:38:00.031378 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:38:00.031340 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:38:00.031857 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:38:00.031466 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:38:00.032212 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:38:00.032167 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:38:00.032335 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:38:00.032318 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:38:00.712671 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:38:00.712642 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:38:02.013775 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:38:02.013745 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:39:03.425662 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:03.425631 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx"] Sep 10 09:39:03.426146 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:03.425841 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="online" containerID="cri-o://a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8" gracePeriod=30 Sep 10 09:39:03.426146 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:03.425890 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="offline" containerID="cri-o://f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16" gracePeriod=30 Sep 10 09:39:03.426146 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:03.425909 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="ui" containerID="cri-o://049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe" gracePeriod=30 Sep 10 09:39:03.567389 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:03.567353 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6"] Sep 10 09:39:03.571017 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:03.570857 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="registry" containerID="cri-o://a3dc9efa8e204fad167ade9a47acc54a30ffcfd741f42ad1fefa7e4573e4057e" gracePeriod=30 Sep 10 09:39:03.571017 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:03.570879 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="offline" containerID="cri-o://a2f990ba522745c98a0cd8cced3c00ccb34622e9fe20aef06b47d55caed27c0f" gracePeriod=30 Sep 10 09:39:03.571017 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:03.570914 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="online" containerID="cri-o://006205b98adc481a822f10642e2c8be2e479e0b818073e812442c2734171dec3" gracePeriod=30 Sep 10 09:39:03.571017 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:03.570925 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="ui" containerID="cri-o://a21c5543cd93e0d6918c5ce065a1fd0bcbfcdfd0c8a2e619221fcd288769daea" gracePeriod=30 Sep 10 09:39:04.577801 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:04.577729 2546 generic.go:356] "Generic (PLEG): container finished" podID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerID="a21c5543cd93e0d6918c5ce065a1fd0bcbfcdfd0c8a2e619221fcd288769daea" exitCode=0 Sep 10 09:39:04.577801 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:04.577754 2546 generic.go:356] "Generic (PLEG): container finished" podID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerID="006205b98adc481a822f10642e2c8be2e479e0b818073e812442c2734171dec3" exitCode=0 Sep 10 09:39:04.578246 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:04.577799 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerDied","Data":"a21c5543cd93e0d6918c5ce065a1fd0bcbfcdfd0c8a2e619221fcd288769daea"} Sep 10 09:39:04.578246 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:04.577841 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerDied","Data":"006205b98adc481a822f10642e2c8be2e479e0b818073e812442c2734171dec3"} Sep 10 09:39:04.580256 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:04.580236 2546 generic.go:356] "Generic (PLEG): container finished" podID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerID="049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe" exitCode=0 Sep 10 09:39:04.580256 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:04.580254 2546 generic.go:356] "Generic (PLEG): container finished" podID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerID="a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8" exitCode=0 Sep 10 09:39:04.580425 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:04.580305 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerDied","Data":"049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe"} Sep 10 09:39:04.580425 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:04.580339 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerDied","Data":"a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8"} Sep 10 09:39:05.125414 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:05.125376 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:39:05.189191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:05.189158 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:39:08.731012 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:08.730977 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:39:09.399782 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:09.399741 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Sep 10 09:39:09.916371 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:09.916302 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 10 09:39:10.032354 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:10.032312 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 10 09:39:11.339414 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:11.339376 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 10 09:39:19.399683 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:19.399638 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Sep 10 09:39:19.915537 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:19.915493 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 10 09:39:20.032114 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:20.032074 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 10 09:39:21.339759 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:21.339726 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 10 09:39:29.399623 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:29.399568 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Sep 10 09:39:29.400080 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:29.399687 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:39:29.915874 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:29.915825 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 10 09:39:29.916060 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:29.915945 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:39:30.032419 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:30.032384 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 10 09:39:30.032561 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:30.032484 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:39:31.339617 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:31.339576 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 10 09:39:31.340015 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:31.339692 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:39:33.619136 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.619114 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:39:33.690311 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.690225 2546 generic.go:356] "Generic (PLEG): container finished" podID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerID="f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16" exitCode=137 Sep 10 09:39:33.690417 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.690313 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" Sep 10 09:39:33.690417 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.690321 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerDied","Data":"f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16"} Sep 10 09:39:33.690417 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.690367 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx" event={"ID":"8ed0a908-6413-4795-99a8-7d6aa4a6b12a","Type":"ContainerDied","Data":"9a7398f19ce4f57a84495b5c2eaaefebceda619cdb2e199cc6c4f5bc44e72673"} Sep 10 09:39:33.690417 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.690386 2546 scope.go:117] "RemoveContainer" containerID="049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe" Sep 10 09:39:33.693546 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.693527 2546 generic.go:356] "Generic (PLEG): container finished" podID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerID="a2f990ba522745c98a0cd8cced3c00ccb34622e9fe20aef06b47d55caed27c0f" exitCode=137 Sep 10 09:39:33.693546 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.693545 2546 generic.go:356] "Generic (PLEG): container finished" podID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerID="a3dc9efa8e204fad167ade9a47acc54a30ffcfd741f42ad1fefa7e4573e4057e" exitCode=137 Sep 10 09:39:33.693696 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.693564 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerDied","Data":"a2f990ba522745c98a0cd8cced3c00ccb34622e9fe20aef06b47d55caed27c0f"} Sep 10 09:39:33.693696 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.693588 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerDied","Data":"a3dc9efa8e204fad167ade9a47acc54a30ffcfd741f42ad1fefa7e4573e4057e"} Sep 10 09:39:33.697994 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.697978 2546 scope.go:117] "RemoveContainer" containerID="f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16" Sep 10 09:39:33.704809 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.704792 2546 scope.go:117] "RemoveContainer" containerID="a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8" Sep 10 09:39:33.711519 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.711504 2546 scope.go:117] "RemoveContainer" containerID="f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33" Sep 10 09:39:33.717779 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.717763 2546 scope.go:117] "RemoveContainer" containerID="fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29" Sep 10 09:39:33.723936 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.723917 2546 scope.go:117] "RemoveContainer" containerID="049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe" Sep 10 09:39:33.724176 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:39:33.724156 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe\": container with ID starting with 049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe not found: ID does not exist" containerID="049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe" Sep 10 09:39:33.724219 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.724185 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe"} err="failed to get container status \"049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe\": rpc error: code = NotFound desc = could not find container \"049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe\": container with ID starting with 049206cf1953704a3f947b9482da0c2d6b1b4ba07fd97e411c1ca4351a32b2fe not found: ID does not exist" Sep 10 09:39:33.724219 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.724201 2546 scope.go:117] "RemoveContainer" containerID="f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16" Sep 10 09:39:33.724447 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:39:33.724431 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16\": container with ID starting with f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16 not found: ID does not exist" containerID="f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16" Sep 10 09:39:33.724495 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.724450 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16"} err="failed to get container status \"f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16\": rpc error: code = NotFound desc = could not find container \"f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16\": container with ID starting with f4ae966aa5c852b27751d9153bc4942f9e6d2610a9c3f80daa24a88cc9e0da16 not found: ID does not exist" Sep 10 09:39:33.724495 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.724465 2546 scope.go:117] "RemoveContainer" containerID="a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8" Sep 10 09:39:33.724694 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:39:33.724680 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8\": container with ID starting with a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8 not found: ID does not exist" containerID="a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8" Sep 10 09:39:33.724739 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.724697 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8"} err="failed to get container status \"a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8\": rpc error: code = NotFound desc = could not find container \"a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8\": container with ID starting with a11b80aa04441cc1902bd4e7c5312af1e29d17fff82f742b032bf90ce7c7d3c8 not found: ID does not exist" Sep 10 09:39:33.724739 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.724709 2546 scope.go:117] "RemoveContainer" containerID="f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33" Sep 10 09:39:33.724938 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:39:33.724923 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33\": container with ID starting with f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33 not found: ID does not exist" containerID="f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33" Sep 10 09:39:33.724983 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.724941 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33"} err="failed to get container status \"f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33\": rpc error: code = NotFound desc = could not find container \"f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33\": container with ID starting with f91f722815deac7c292ac65cf31e069c9a9d850dc73404ecf768e85d2838bf33 not found: ID does not exist" Sep 10 09:39:33.724983 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.724953 2546 scope.go:117] "RemoveContainer" containerID="fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29" Sep 10 09:39:33.725134 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:39:33.725120 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29\": container with ID starting with fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29 not found: ID does not exist" containerID="fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29" Sep 10 09:39:33.725175 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.725136 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29"} err="failed to get container status \"fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29\": rpc error: code = NotFound desc = could not find container \"fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29\": container with ID starting with fa41574354bcac8712bb2db372a624d45222c0439d840d9cb005b6e8a8aeac29 not found: ID does not exist" Sep 10 09:39:33.742429 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.742410 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-registry-tls\") pod \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " Sep 10 09:39:33.742496 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.742435 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-offline-tls\") pod \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " Sep 10 09:39:33.742496 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.742455 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-online-tls\") pod \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " Sep 10 09:39:33.742496 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.742493 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-ui-tls\") pod \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " Sep 10 09:39:33.742600 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.742509 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-feast-data\") pod \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " Sep 10 09:39:33.742600 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.742544 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbmmv\" (UniqueName: \"kubernetes.io/projected/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-kube-api-access-qbmmv\") pod \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\" (UID: \"8ed0a908-6413-4795-99a8-7d6aa4a6b12a\") " Sep 10 09:39:33.742796 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.742773 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8ed0a908-6413-4795-99a8-7d6aa4a6b12a" (UID: "8ed0a908-6413-4795-99a8-7d6aa4a6b12a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:39:33.743021 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.742991 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-feast-data" (OuterVolumeSpecName: "feast-data") pod "8ed0a908-6413-4795-99a8-7d6aa4a6b12a" (UID: "8ed0a908-6413-4795-99a8-7d6aa4a6b12a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:39:33.744496 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.744475 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "8ed0a908-6413-4795-99a8-7d6aa4a6b12a" (UID: "8ed0a908-6413-4795-99a8-7d6aa4a6b12a"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:39:33.744574 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.744493 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-online-tls" (OuterVolumeSpecName: "online-tls") pod "8ed0a908-6413-4795-99a8-7d6aa4a6b12a" (UID: "8ed0a908-6413-4795-99a8-7d6aa4a6b12a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:39:33.744574 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.744511 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "8ed0a908-6413-4795-99a8-7d6aa4a6b12a" (UID: "8ed0a908-6413-4795-99a8-7d6aa4a6b12a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:39:33.744574 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.744561 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-kube-api-access-qbmmv" (OuterVolumeSpecName: "kube-api-access-qbmmv") pod "8ed0a908-6413-4795-99a8-7d6aa4a6b12a" (UID: "8ed0a908-6413-4795-99a8-7d6aa4a6b12a"). InnerVolumeSpecName "kube-api-access-qbmmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:39:33.844048 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.844022 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbmmv\" (UniqueName: \"kubernetes.io/projected/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-kube-api-access-qbmmv\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:33.844048 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.844046 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:33.844161 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.844055 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:33.844161 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.844062 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:33.844161 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.844070 2546 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-ui-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:33.844161 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.844078 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed0a908-6413-4795-99a8-7d6aa4a6b12a-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:33.927102 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:33.927077 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:39:34.011777 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.011756 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx"] Sep 10 09:39:34.015238 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.015218 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-6wdxx"] Sep 10 09:39:34.207468 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.207443 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:39:34.247720 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.247694 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-registry-tls\") pod \"9a8ad248-5f07-428d-aa29-16f330c2f449\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " Sep 10 09:39:34.247836 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.247733 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-ui-tls\") pod \"9a8ad248-5f07-428d-aa29-16f330c2f449\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " Sep 10 09:39:34.247836 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.247769 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-offline-tls\") pod \"9a8ad248-5f07-428d-aa29-16f330c2f449\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " Sep 10 09:39:34.247836 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.247787 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8ad248-5f07-428d-aa29-16f330c2f449-feast-data\") pod \"9a8ad248-5f07-428d-aa29-16f330c2f449\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " Sep 10 09:39:34.247836 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.247823 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-online-tls\") pod \"9a8ad248-5f07-428d-aa29-16f330c2f449\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " Sep 10 09:39:34.248049 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.247860 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znxc4\" (UniqueName: \"kubernetes.io/projected/9a8ad248-5f07-428d-aa29-16f330c2f449-kube-api-access-znxc4\") pod \"9a8ad248-5f07-428d-aa29-16f330c2f449\" (UID: \"9a8ad248-5f07-428d-aa29-16f330c2f449\") " Sep 10 09:39:34.248438 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.248370 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8ad248-5f07-428d-aa29-16f330c2f449-feast-data" (OuterVolumeSpecName: "feast-data") pod "9a8ad248-5f07-428d-aa29-16f330c2f449" (UID: "9a8ad248-5f07-428d-aa29-16f330c2f449"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:39:34.249861 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.249830 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "9a8ad248-5f07-428d-aa29-16f330c2f449" (UID: "9a8ad248-5f07-428d-aa29-16f330c2f449"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:39:34.249956 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.249892 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9a8ad248-5f07-428d-aa29-16f330c2f449" (UID: "9a8ad248-5f07-428d-aa29-16f330c2f449"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:39:34.249956 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.249908 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-online-tls" (OuterVolumeSpecName: "online-tls") pod "9a8ad248-5f07-428d-aa29-16f330c2f449" (UID: "9a8ad248-5f07-428d-aa29-16f330c2f449"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:39:34.250191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.250159 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9a8ad248-5f07-428d-aa29-16f330c2f449" (UID: "9a8ad248-5f07-428d-aa29-16f330c2f449"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:39:34.250191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.250162 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8ad248-5f07-428d-aa29-16f330c2f449-kube-api-access-znxc4" (OuterVolumeSpecName: "kube-api-access-znxc4") pod "9a8ad248-5f07-428d-aa29-16f330c2f449" (UID: "9a8ad248-5f07-428d-aa29-16f330c2f449"). InnerVolumeSpecName "kube-api-access-znxc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:39:34.348552 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.348531 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:34.348552 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.348549 2546 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-ui-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:34.348679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.348557 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:34.348679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.348565 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8ad248-5f07-428d-aa29-16f330c2f449-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:34.348679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.348573 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a8ad248-5f07-428d-aa29-16f330c2f449-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:34.348679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.348582 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxc4\" (UniqueName: \"kubernetes.io/projected/9a8ad248-5f07-428d-aa29-16f330c2f449-kube-api-access-znxc4\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:39:34.701868 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.701836 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" Sep 10 09:39:34.702361 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.701837 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6" event={"ID":"9a8ad248-5f07-428d-aa29-16f330c2f449","Type":"ContainerDied","Data":"4ebbcdb0eb875513990515de067ac7e8d1d601b73d22e6956a3b47b1bc182dc7"} Sep 10 09:39:34.702361 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.701968 2546 scope.go:117] "RemoveContainer" containerID="a21c5543cd93e0d6918c5ce065a1fd0bcbfcdfd0c8a2e619221fcd288769daea" Sep 10 09:39:34.709618 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.709597 2546 scope.go:117] "RemoveContainer" containerID="a2f990ba522745c98a0cd8cced3c00ccb34622e9fe20aef06b47d55caed27c0f" Sep 10 09:39:34.717407 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.717388 2546 scope.go:117] "RemoveContainer" containerID="006205b98adc481a822f10642e2c8be2e479e0b818073e812442c2734171dec3" Sep 10 09:39:34.723852 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.723826 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6"] Sep 10 09:39:34.724021 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:39:34.723994 2546 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a8ad248_5f07_428d_aa29_16f330c2f449.slice/crio-4ebbcdb0eb875513990515de067ac7e8d1d601b73d22e6956a3b47b1bc182dc7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a8ad248_5f07_428d_aa29_16f330c2f449.slice\": RecentStats: unable to find data in memory cache]" Sep 10 09:39:34.725879 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.725865 2546 scope.go:117] "RemoveContainer" containerID="a3dc9efa8e204fad167ade9a47acc54a30ffcfd741f42ad1fefa7e4573e4057e" Sep 10 09:39:34.729729 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.729708 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-qcbr6"] Sep 10 09:39:34.738083 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.738055 2546 scope.go:117] "RemoveContainer" containerID="5b2659207a11afac6f1ddafa2ee7b05c9313bfe2caeee1902b62e52efcdaf5c1" Sep 10 09:39:34.744531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:34.744516 2546 scope.go:117] "RemoveContainer" containerID="2d5726d3258da6850d57f6df2a3c99998fd4b57bb17f2d7aaec42567b646c735" Sep 10 09:39:35.260598 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:35.260568 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" path="/var/lib/kubelet/pods/8ed0a908-6413-4795-99a8-7d6aa4a6b12a/volumes" Sep 10 09:39:35.261161 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:35.261145 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" path="/var/lib/kubelet/pods/9a8ad248-5f07-428d-aa29-16f330c2f449/volumes" Sep 10 09:39:46.571084 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:46.570979 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:39:46.677556 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:46.677523 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:39:52.256018 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:52.255975 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:39:52.316142 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:39:52.316107 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:40:13.139633 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:13.139588 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:40:13.240162 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:13.240127 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:40:15.016214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.016185 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:40:15.116299 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.116242 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:40:15.119826 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.119796 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g"] Sep 10 09:40:15.120104 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120091 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="ui" Sep 10 09:40:15.120152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120106 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="ui" Sep 10 09:40:15.120152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120118 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="feast-init" Sep 10 09:40:15.120152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120124 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="feast-init" Sep 10 09:40:15.120152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120136 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="feast-init" Sep 10 09:40:15.120152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120146 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="feast-init" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120158 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="online" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120163 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="online" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120170 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="offline" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120175 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="offline" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120181 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="ui" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120186 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="ui" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120192 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="feast-apply" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120197 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="feast-apply" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120203 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="online" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120208 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="online" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120220 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="registry" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120226 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="registry" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120232 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="offline" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120237 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="offline" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120243 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="feast-apply" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120248 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="feast-apply" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120302 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="online" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120310 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="offline" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120316 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="registry" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120323 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="offline" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120328 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed0a908-6413-4795-99a8-7d6aa4a6b12a" containerName="ui" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120333 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="online" Sep 10 09:40:15.120340 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.120340 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8ad248-5f07-428d-aa29-16f330c2f449" containerName="ui" Sep 10 09:40:15.123647 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.123630 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.125529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.125508 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 10 09:40:15.125694 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.125674 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 10 09:40:15.125837 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.125823 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 10 09:40:15.125898 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.125829 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 10 09:40:15.125898 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.125857 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 10 09:40:15.126316 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.126299 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-nq6pz\"" Sep 10 09:40:15.126414 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.126314 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 10 09:40:15.134652 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.134634 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g"] Sep 10 09:40:15.171413 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.171386 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:40:15.240707 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.240684 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e3c9-2603-4501-8dae-b650086c2bbe-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.240794 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.240718 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.240794 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.240753 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.240868 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.240793 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppf7b\" (UniqueName: \"kubernetes.io/projected/d2d6e3c9-2603-4501-8dae-b650086c2bbe-kube-api-access-ppf7b\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.240868 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.240827 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.240868 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.240855 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.303117 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.303057 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:40:15.342121 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.342094 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.342220 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.342162 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e3c9-2603-4501-8dae-b650086c2bbe-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.342220 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.342198 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.342359 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:40:15.342223 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 10 09:40:15.342359 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.342226 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.342359 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:40:15.342296 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-offline-tls podName:d2d6e3c9-2603-4501-8dae-b650086c2bbe nodeName:}" failed. No retries permitted until 2026-09-10 09:40:15.842257413 +0000 UTC m=+1555.023946457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-74p7g" (UID: "d2d6e3c9-2603-4501-8dae-b650086c2bbe") : secret "feast-simple-feast-setup-offline-tls" not found Sep 10 09:40:15.342359 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:40:15.342321 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 10 09:40:15.342359 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.342342 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppf7b\" (UniqueName: \"kubernetes.io/projected/d2d6e3c9-2603-4501-8dae-b650086c2bbe-kube-api-access-ppf7b\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.342612 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:40:15.342367 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-registry-tls podName:d2d6e3c9-2603-4501-8dae-b650086c2bbe nodeName:}" failed. No retries permitted until 2026-09-10 09:40:15.842354753 +0000 UTC m=+1555.024043796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-74p7g" (UID: "d2d6e3c9-2603-4501-8dae-b650086c2bbe") : secret "feast-simple-feast-setup-registry-tls" not found Sep 10 09:40:15.342612 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.342394 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.342612 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.342591 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e3c9-2603-4501-8dae-b650086c2bbe-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.344743 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.344723 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.344743 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.344736 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.351429 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.351411 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppf7b\" (UniqueName: \"kubernetes.io/projected/d2d6e3c9-2603-4501-8dae-b650086c2bbe-kube-api-access-ppf7b\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.719569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.719475 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:40:15.819481 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.819453 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:40:15.846426 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.846396 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.846552 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.846453 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.848721 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.848697 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:15.848815 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:15.848725 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-74p7g\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:16.048804 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:16.048776 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:16.320930 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:16.320906 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g"] Sep 10 09:40:16.322590 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:40:16.322570 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2d6e3c9_2603_4501_8dae_b650086c2bbe.slice/crio-40f34e4aaf09c37b94b95cfbdd1d4edb7880868c4ec7ee94f19a5a5f5d595ea7 WatchSource:0}: Error finding container 40f34e4aaf09c37b94b95cfbdd1d4edb7880868c4ec7ee94f19a5a5f5d595ea7: Status 404 returned error can't find the container with id 40f34e4aaf09c37b94b95cfbdd1d4edb7880868c4ec7ee94f19a5a5f5d595ea7 Sep 10 09:40:16.850017 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:16.849979 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerStarted","Data":"6261c39008e6cbc662dc1f445383514676fabe9bf58abaae6da277a34781abed"} Sep 10 09:40:16.850017 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:16.850020 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerStarted","Data":"40f34e4aaf09c37b94b95cfbdd1d4edb7880868c4ec7ee94f19a5a5f5d595ea7"} Sep 10 09:40:21.871324 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:21.871267 2546 generic.go:356] "Generic (PLEG): container finished" podID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerID="6261c39008e6cbc662dc1f445383514676fabe9bf58abaae6da277a34781abed" exitCode=0 Sep 10 09:40:21.871324 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:21.871326 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerDied","Data":"6261c39008e6cbc662dc1f445383514676fabe9bf58abaae6da277a34781abed"} Sep 10 09:40:22.876479 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:22.876443 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerStarted","Data":"fb5334d3b1569f198b5a6df0a76f1b3b7d1850abd211294b9d69576a542bc2e7"} Sep 10 09:40:27.897626 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:27.897592 2546 generic.go:356] "Generic (PLEG): container finished" podID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerID="fb5334d3b1569f198b5a6df0a76f1b3b7d1850abd211294b9d69576a542bc2e7" exitCode=0 Sep 10 09:40:27.898137 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:27.897674 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerDied","Data":"fb5334d3b1569f198b5a6df0a76f1b3b7d1850abd211294b9d69576a542bc2e7"} Sep 10 09:40:28.907190 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:28.907135 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerStarted","Data":"e997fc18d4365d9413be2eac34ea8c11e7a80ab73710218b5df06cfc0133a1b7"} Sep 10 09:40:28.907704 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:28.907210 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerStarted","Data":"9ad9594f3c45d0e794573440b6a9b64c193a69143594f461677a3401c5214054"} Sep 10 09:40:28.907704 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:28.907234 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerStarted","Data":"b989c56846a46b91340c19a4c0d20ee80dc18746da2bad277ac0f02e7f0857f1"} Sep 10 09:40:28.907704 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:28.907252 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerStarted","Data":"2c9a7eed98197d4c1c140f63018714f5a3b2ca928fa491630a227c270d730109"} Sep 10 09:40:28.930419 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:28.930359 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podStartSLOduration=13.930337987 podStartE2EDuration="13.930337987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:40:28.927415519 +0000 UTC m=+1568.109104585" watchObservedRunningTime="2026-09-10 09:40:28.930337987 +0000 UTC m=+1568.112027054" Sep 10 09:40:31.049802 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:31.049767 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:31.050368 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:31.050344 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:31.050510 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:31.050375 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:31.050510 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:31.050393 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:31.051845 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:31.051818 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 10 09:40:31.052047 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:31.051870 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 10 09:40:31.052129 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:31.051906 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused" Sep 10 09:40:31.052129 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:31.051919 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused" Sep 10 09:40:34.049762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:34.049722 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused" Sep 10 09:40:34.050357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:34.049722 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused" Sep 10 09:40:34.050357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:34.049768 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 10 09:40:34.050558 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:34.049890 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 10 09:40:36.049219 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:36.049184 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:36.049593 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:36.049239 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:36.049593 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:36.049254 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:36.049593 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:36.049264 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:37.050546 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:37.050517 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:37.050986 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:37.050734 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:37.051051 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:37.050991 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:37.051442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:37.051420 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:37.051587 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:37.051517 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:37.051587 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:37.051570 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:37.056262 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:37.056243 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:40:37.953763 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:40:37.953738 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:41:39.761019 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:39.760985 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:41:39.766001 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:39.765976 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g"] Sep 10 09:41:39.766224 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:39.766199 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="registry" containerID="cri-o://2c9a7eed98197d4c1c140f63018714f5a3b2ca928fa491630a227c270d730109" gracePeriod=30 Sep 10 09:41:39.766348 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:39.766267 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="online" containerID="cri-o://b989c56846a46b91340c19a4c0d20ee80dc18746da2bad277ac0f02e7f0857f1" gracePeriod=30 Sep 10 09:41:39.766417 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:39.766264 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="ui" containerID="cri-o://e997fc18d4365d9413be2eac34ea8c11e7a80ab73710218b5df06cfc0133a1b7" gracePeriod=30 Sep 10 09:41:39.766966 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:39.766530 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="offline" containerID="cri-o://9ad9594f3c45d0e794573440b6a9b64c193a69143594f461677a3401c5214054" gracePeriod=30 Sep 10 09:41:40.151497 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.151401 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4"] Sep 10 09:41:40.155363 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.155340 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.157599 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.157573 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-nhknw\"" Sep 10 09:41:40.168236 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.168208 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4"] Sep 10 09:41:40.289554 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.289526 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/173b7933-7291-40d1-b33c-dc19066eef19-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.289692 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.289560 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4df\" (UniqueName: \"kubernetes.io/projected/173b7933-7291-40d1-b33c-dc19066eef19-kube-api-access-9g4df\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.289692 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.289660 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.289794 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.289695 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.289794 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.289723 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.289865 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.289800 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.390784 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.390749 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.390968 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:41:40.390912 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 10 09:41:40.390968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.390932 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/173b7933-7291-40d1-b33c-dc19066eef19-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.391073 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:41:40.390981 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-registry-tls podName:173b7933-7291-40d1-b33c-dc19066eef19 nodeName:}" failed. No retries permitted until 2026-09-10 09:41:40.890960077 +0000 UTC m=+1640.072649122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-4v5g4" (UID: "173b7933-7291-40d1-b33c-dc19066eef19") : secret "feast-simple-feast-setup-registry-tls" not found Sep 10 09:41:40.391073 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.391008 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4df\" (UniqueName: \"kubernetes.io/projected/173b7933-7291-40d1-b33c-dc19066eef19-kube-api-access-9g4df\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.391073 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.391064 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.391321 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.391084 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.391321 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.391106 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.391321 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:41:40.391231 2546 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 10 09:41:40.391321 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:41:40.391271 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-online-tls podName:173b7933-7291-40d1-b33c-dc19066eef19 nodeName:}" failed. No retries permitted until 2026-09-10 09:41:40.891258747 +0000 UTC m=+1640.072947795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-online-tls") pod "feast-simple-feast-setup-57576b8fcf-4v5g4" (UID: "173b7933-7291-40d1-b33c-dc19066eef19") : secret "feast-simple-feast-setup-online-tls" not found Sep 10 09:41:40.391473 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.391410 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/173b7933-7291-40d1-b33c-dc19066eef19-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.393512 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.393490 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.393620 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.393543 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.401767 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.401705 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4df\" (UniqueName: \"kubernetes.io/projected/173b7933-7291-40d1-b33c-dc19066eef19-kube-api-access-9g4df\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.895679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.895639 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.896021 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.895725 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.898035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.898012 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:40.898035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:40.898027 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-4v5g4\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:41.064208 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:41.064180 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:41.185793 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:41.185720 2546 generic.go:356] "Generic (PLEG): container finished" podID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerID="e997fc18d4365d9413be2eac34ea8c11e7a80ab73710218b5df06cfc0133a1b7" exitCode=0 Sep 10 09:41:41.185793 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:41.185745 2546 generic.go:356] "Generic (PLEG): container finished" podID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerID="b989c56846a46b91340c19a4c0d20ee80dc18746da2bad277ac0f02e7f0857f1" exitCode=0 Sep 10 09:41:41.185935 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:41.185787 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerDied","Data":"e997fc18d4365d9413be2eac34ea8c11e7a80ab73710218b5df06cfc0133a1b7"} Sep 10 09:41:41.185935 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:41.185828 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerDied","Data":"b989c56846a46b91340c19a4c0d20ee80dc18746da2bad277ac0f02e7f0857f1"} Sep 10 09:41:41.337075 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:41.337052 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4"] Sep 10 09:41:41.338764 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:41:41.338740 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod173b7933_7291_40d1_b33c_dc19066eef19.slice/crio-cad281ad8396123d8abc8f872d2fb9d41123995990fa04e73b538a99f5f04db1 WatchSource:0}: Error finding container cad281ad8396123d8abc8f872d2fb9d41123995990fa04e73b538a99f5f04db1: Status 404 returned error can't find the container with id cad281ad8396123d8abc8f872d2fb9d41123995990fa04e73b538a99f5f04db1 Sep 10 09:41:41.423487 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:41.423459 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:41:42.190883 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:42.190850 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerStarted","Data":"a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007"} Sep 10 09:41:42.191251 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:42.190892 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerStarted","Data":"cad281ad8396123d8abc8f872d2fb9d41123995990fa04e73b538a99f5f04db1"} Sep 10 09:41:46.208936 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:46.208899 2546 generic.go:356] "Generic (PLEG): container finished" podID="173b7933-7291-40d1-b33c-dc19066eef19" containerID="a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007" exitCode=0 Sep 10 09:41:46.209297 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:46.208945 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerDied","Data":"a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007"} Sep 10 09:41:47.051347 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:47.051297 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 10 09:41:47.215931 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:47.215900 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerStarted","Data":"897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4"} Sep 10 09:41:47.951064 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:47.951026 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 10 09:41:52.238566 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:52.238529 2546 generic.go:356] "Generic (PLEG): container finished" podID="173b7933-7291-40d1-b33c-dc19066eef19" containerID="897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4" exitCode=0 Sep 10 09:41:52.238915 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:52.238599 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerDied","Data":"897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4"} Sep 10 09:41:53.248212 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:53.248171 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerStarted","Data":"bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0"} Sep 10 09:41:53.248718 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:53.248235 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerStarted","Data":"c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506"} Sep 10 09:41:53.248718 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:53.248256 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerStarted","Data":"09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3"} Sep 10 09:41:53.248718 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:53.248274 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerStarted","Data":"77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2"} Sep 10 09:41:53.270848 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:53.270797 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podStartSLOduration=13.27077588 podStartE2EDuration="13.27077588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:41:53.26923426 +0000 UTC m=+1652.450923337" watchObservedRunningTime="2026-09-10 09:41:53.27077588 +0000 UTC m=+1652.452464947" Sep 10 09:41:56.065088 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:56.065055 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:56.065677 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:56.065651 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:56.065823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:56.065688 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:56.065823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:56.065707 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:41:56.067532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:56.067401 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused" Sep 10 09:41:56.067532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:56.067435 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 10 09:41:56.067532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:56.067471 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Sep 10 09:41:56.067532 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:56.067439 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 10 09:41:57.051578 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:57.051525 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 10 09:41:57.950920 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:57.950854 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 10 09:41:59.064968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:59.064916 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused" Sep 10 09:41:59.064968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:59.064957 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 10 09:41:59.065520 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:59.064925 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 10 09:41:59.065520 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:41:59.064912 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Sep 10 09:42:01.065198 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:01.065150 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:01.065601 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:01.065216 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:01.065601 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:01.065230 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:01.065601 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:01.065241 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:02.065732 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:02.065697 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:02.066212 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:02.065951 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:02.066477 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:02.066454 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:02.066549 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:02.066528 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:02.066592 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:02.066576 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:02.071588 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:02.071569 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:02.292446 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:02.292419 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:02.295257 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:02.295231 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:42:07.051859 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:07.051812 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 10 09:42:07.052390 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:07.051961 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:42:07.951594 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:07.951553 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 10 09:42:07.951772 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:07.951655 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:42:10.325097 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.325064 2546 generic.go:356] "Generic (PLEG): container finished" podID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerID="9ad9594f3c45d0e794573440b6a9b64c193a69143594f461677a3401c5214054" exitCode=137 Sep 10 09:42:10.325097 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.325091 2546 generic.go:356] "Generic (PLEG): container finished" podID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerID="2c9a7eed98197d4c1c140f63018714f5a3b2ca928fa491630a227c270d730109" exitCode=137 Sep 10 09:42:10.325612 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.325135 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerDied","Data":"9ad9594f3c45d0e794573440b6a9b64c193a69143594f461677a3401c5214054"} Sep 10 09:42:10.325612 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.325174 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerDied","Data":"2c9a7eed98197d4c1c140f63018714f5a3b2ca928fa491630a227c270d730109"} Sep 10 09:42:10.408197 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.408170 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:42:10.445877 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.445842 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-offline-tls\") pod \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " Sep 10 09:42:10.446010 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.445883 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-online-tls\") pod \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " Sep 10 09:42:10.446010 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.445993 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppf7b\" (UniqueName: \"kubernetes.io/projected/d2d6e3c9-2603-4501-8dae-b650086c2bbe-kube-api-access-ppf7b\") pod \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " Sep 10 09:42:10.446098 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.446053 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-registry-tls\") pod \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " Sep 10 09:42:10.446098 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.446081 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e3c9-2603-4501-8dae-b650086c2bbe-feast-data\") pod \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " Sep 10 09:42:10.446180 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.446105 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-ui-tls\") pod \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\" (UID: \"d2d6e3c9-2603-4501-8dae-b650086c2bbe\") " Sep 10 09:42:10.446678 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.446649 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d6e3c9-2603-4501-8dae-b650086c2bbe-feast-data" (OuterVolumeSpecName: "feast-data") pod "d2d6e3c9-2603-4501-8dae-b650086c2bbe" (UID: "d2d6e3c9-2603-4501-8dae-b650086c2bbe"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:42:10.448270 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.448243 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d2d6e3c9-2603-4501-8dae-b650086c2bbe" (UID: "d2d6e3c9-2603-4501-8dae-b650086c2bbe"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:42:10.448270 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.448262 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d2d6e3c9-2603-4501-8dae-b650086c2bbe" (UID: "d2d6e3c9-2603-4501-8dae-b650086c2bbe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:42:10.448455 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.448253 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d6e3c9-2603-4501-8dae-b650086c2bbe-kube-api-access-ppf7b" (OuterVolumeSpecName: "kube-api-access-ppf7b") pod "d2d6e3c9-2603-4501-8dae-b650086c2bbe" (UID: "d2d6e3c9-2603-4501-8dae-b650086c2bbe"). InnerVolumeSpecName "kube-api-access-ppf7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:42:10.448455 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.448408 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d2d6e3c9-2603-4501-8dae-b650086c2bbe" (UID: "d2d6e3c9-2603-4501-8dae-b650086c2bbe"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:42:10.448455 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.448428 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-online-tls" (OuterVolumeSpecName: "online-tls") pod "d2d6e3c9-2603-4501-8dae-b650086c2bbe" (UID: "d2d6e3c9-2603-4501-8dae-b650086c2bbe"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:42:10.547565 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.547501 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppf7b\" (UniqueName: \"kubernetes.io/projected/d2d6e3c9-2603-4501-8dae-b650086c2bbe-kube-api-access-ppf7b\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:42:10.547565 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.547533 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:42:10.547565 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.547542 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d6e3c9-2603-4501-8dae-b650086c2bbe-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:42:10.547565 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.547551 2546 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-ui-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:42:10.547565 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.547558 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:42:10.547565 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:10.547565 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d2d6e3c9-2603-4501-8dae-b650086c2bbe-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:42:11.331629 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.331603 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" Sep 10 09:42:11.331629 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.331615 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g" event={"ID":"d2d6e3c9-2603-4501-8dae-b650086c2bbe","Type":"ContainerDied","Data":"40f34e4aaf09c37b94b95cfbdd1d4edb7880868c4ec7ee94f19a5a5f5d595ea7"} Sep 10 09:42:11.332045 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.331665 2546 scope.go:117] "RemoveContainer" containerID="e997fc18d4365d9413be2eac34ea8c11e7a80ab73710218b5df06cfc0133a1b7" Sep 10 09:42:11.341899 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.341881 2546 scope.go:117] "RemoveContainer" containerID="9ad9594f3c45d0e794573440b6a9b64c193a69143594f461677a3401c5214054" Sep 10 09:42:11.349665 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.349644 2546 scope.go:117] "RemoveContainer" containerID="b989c56846a46b91340c19a4c0d20ee80dc18746da2bad277ac0f02e7f0857f1" Sep 10 09:42:11.352785 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.351275 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g"] Sep 10 09:42:11.355837 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.355806 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-74p7g"] Sep 10 09:42:11.363545 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.363526 2546 scope.go:117] "RemoveContainer" containerID="2c9a7eed98197d4c1c140f63018714f5a3b2ca928fa491630a227c270d730109" Sep 10 09:42:11.370560 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.370542 2546 scope.go:117] "RemoveContainer" containerID="fb5334d3b1569f198b5a6df0a76f1b3b7d1850abd211294b9d69576a542bc2e7" Sep 10 09:42:11.377051 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:11.377033 2546 scope.go:117] "RemoveContainer" containerID="6261c39008e6cbc662dc1f445383514676fabe9bf58abaae6da277a34781abed" Sep 10 09:42:13.260569 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:13.260537 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" path="/var/lib/kubelet/pods/d2d6e3c9-2603-4501-8dae-b650086c2bbe/volumes" Sep 10 09:42:50.784595 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:42:50.784493 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:04.297199 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.297163 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:04.397251 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.397219 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:04.689215 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.689132 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:04.770018 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.769989 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:04.776892 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.776867 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt"] Sep 10 09:43:04.777146 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777136 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="feast-apply" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777148 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="feast-apply" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777156 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="offline" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777162 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="offline" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777171 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="registry" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777176 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="registry" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777183 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="ui" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777188 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="ui" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777198 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="feast-init" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777203 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="feast-init" Sep 10 09:43:04.777214 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777214 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="online" Sep 10 09:43:04.777521 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777220 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="online" Sep 10 09:43:04.777521 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777266 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="ui" Sep 10 09:43:04.777521 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777273 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="registry" Sep 10 09:43:04.777521 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777293 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="online" Sep 10 09:43:04.777521 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.777299 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d6e3c9-2603-4501-8dae-b650086c2bbe" containerName="offline" Sep 10 09:43:04.780357 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.780342 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.782243 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.782218 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 10 09:43:04.782371 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.782331 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 10 09:43:04.782559 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.782543 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 10 09:43:04.782664 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.782581 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-zfk82\"" Sep 10 09:43:04.782664 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.782641 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 10 09:43:04.782903 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.782887 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 10 09:43:04.783186 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.783171 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 10 09:43:04.789679 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.789662 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt"] Sep 10 09:43:04.825491 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.825462 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:04.842979 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.842956 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.843068 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.842989 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4nr9\" (UniqueName: \"kubernetes.io/projected/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-kube-api-access-r4nr9\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.843068 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.843019 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.843154 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.843102 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.843154 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.843133 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.843154 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.843151 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.885660 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.885638 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:04.943887 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.943823 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.943887 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.943856 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.943887 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.943882 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.944079 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.943916 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4nr9\" (UniqueName: \"kubernetes.io/projected/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-kube-api-access-r4nr9\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.944079 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.943967 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.944079 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:43:04.944027 2546 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 10 09:43:04.944247 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:43:04.944113 2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-ui-tls podName:2388b8d9-e68e-4b42-9d59-e8d8e69734c3 nodeName:}" failed. No retries permitted until 2026-09-10 09:43:05.444095641 +0000 UTC m=+1724.625784698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" (UID: "2388b8d9-e68e-4b42-9d59-e8d8e69734c3") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 10 09:43:04.944247 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.944028 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.944247 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.944205 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.944660 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.944643 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.946373 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.946354 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.946483 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.946468 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:04.955322 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:04.955303 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4nr9\" (UniqueName: \"kubernetes.io/projected/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-kube-api-access-r4nr9\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:05.273111 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:05.273079 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:05.447817 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:05.447790 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:05.450083 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:05.450063 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-8vtwt\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:05.689601 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:05.689526 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:05.959865 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:05.959841 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt"] Sep 10 09:43:05.961490 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:43:05.961463 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2388b8d9_e68e_4b42_9d59_e8d8e69734c3.slice/crio-255edf4fd25806122ffbd299f5f9bb6ddce87b4ac82f374d38705296777629a4 WatchSource:0}: Error finding container 255edf4fd25806122ffbd299f5f9bb6ddce87b4ac82f374d38705296777629a4: Status 404 returned error can't find the container with id 255edf4fd25806122ffbd299f5f9bb6ddce87b4ac82f374d38705296777629a4 Sep 10 09:43:06.533164 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:06.533128 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerStarted","Data":"9345466b0f26379a8f83718b0826bb9b5f4536bfdd6b4dc656366e7b6beab25a"} Sep 10 09:43:06.533528 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:06.533175 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerStarted","Data":"255edf4fd25806122ffbd299f5f9bb6ddce87b4ac82f374d38705296777629a4"} Sep 10 09:43:11.551499 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:11.551462 2546 generic.go:356] "Generic (PLEG): container finished" podID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerID="9345466b0f26379a8f83718b0826bb9b5f4536bfdd6b4dc656366e7b6beab25a" exitCode=0 Sep 10 09:43:11.551865 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:11.551537 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerDied","Data":"9345466b0f26379a8f83718b0826bb9b5f4536bfdd6b4dc656366e7b6beab25a"} Sep 10 09:43:12.556670 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:12.556637 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerStarted","Data":"58e5da6360fb5abf431e15b69965850b86f43ca9eb682c8302f153da1e053a8e"} Sep 10 09:43:18.119213 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:18.119177 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:30.530473 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:30.530431 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:30.593823 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:30.593790 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:43:32.630952 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:32.630866 2546 generic.go:356] "Generic (PLEG): container finished" podID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerID="58e5da6360fb5abf431e15b69965850b86f43ca9eb682c8302f153da1e053a8e" exitCode=0 Sep 10 09:43:32.630952 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:32.630904 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerDied","Data":"58e5da6360fb5abf431e15b69965850b86f43ca9eb682c8302f153da1e053a8e"} Sep 10 09:43:33.639451 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:33.639417 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerStarted","Data":"a89da2c5e5156a32be57c5271bfa8c18348043d5fbfd745b2bf8478036372589"} Sep 10 09:43:33.639912 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:33.639471 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerStarted","Data":"ea777333dfb4f48c3cf40a41bb572fac5fc93c010e092ac81e7e4f7f6171d0e6"} Sep 10 09:43:33.639912 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:33.639488 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerStarted","Data":"1634933b86634a2c656c6de34e1ed2d103d906205c1ad0d5e35e17a6679b0b64"} Sep 10 09:43:33.658909 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:33.658860 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podStartSLOduration=29.658842741 podStartE2EDuration="29.658842741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:43:33.658138873 +0000 UTC m=+1752.839827939" watchObservedRunningTime="2026-09-10 09:43:33.658842741 +0000 UTC m=+1752.840531807" Sep 10 09:43:35.690462 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:35.690423 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:35.690854 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:35.690488 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:35.690854 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:35.690507 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:35.690854 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:35.690523 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:35.690854 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:35.690539 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:35.690854 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:35.690555 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:35.692621 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:35.692585 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Sep 10 09:43:35.692766 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:35.692656 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 10 09:43:35.692766 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:35.692593 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 10 09:43:38.690584 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:38.690542 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 10 09:43:38.690584 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:38.690573 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 10 09:43:38.691126 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:38.690544 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Sep 10 09:43:41.690539 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:41.690508 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:41.690887 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:41.690610 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:41.691298 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:41.691265 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:41.691415 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:41.691332 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:42.694421 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:42.694384 2546 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 09:43:45.379727 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:45.379693 2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:46.072921 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:46.072893 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:43:46.694450 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:43:46.694400 2546 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 10 09:44:06.694469 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:06.694421 2546 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 10 09:44:26.694406 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:26.694365 2546 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 09:44:46.694418 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:46.694373 2546 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 10 09:44:46.923086 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:46.923049 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:44:46.923719 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:46.923690 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt"] Sep 10 09:44:46.923973 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:46.923943 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" containerID="cri-o://1634933b86634a2c656c6de34e1ed2d103d906205c1ad0d5e35e17a6679b0b64" gracePeriod=30 Sep 10 09:44:46.924183 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:46.923992 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="ui" containerID="cri-o://a89da2c5e5156a32be57c5271bfa8c18348043d5fbfd745b2bf8478036372589" gracePeriod=30 Sep 10 09:44:46.924183 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:46.923980 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="offline" containerID="cri-o://ea777333dfb4f48c3cf40a41bb572fac5fc93c010e092ac81e7e4f7f6171d0e6" gracePeriod=30 Sep 10 09:44:47.032102 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.032068 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:44:47.073536 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.073507 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4"] Sep 10 09:44:47.073771 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.073729 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="registry" containerID="cri-o://77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2" gracePeriod=30 Sep 10 09:44:47.073915 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.073783 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="online" containerID="cri-o://09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3" gracePeriod=30 Sep 10 09:44:47.073915 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.073830 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="ui" containerID="cri-o://bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0" gracePeriod=30 Sep 10 09:44:47.073915 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.073783 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="offline" containerID="cri-o://c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506" gracePeriod=30 Sep 10 09:44:47.123427 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.123389 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:44:47.183785 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.183736 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:44:47.932908 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.932881 2546 generic.go:356] "Generic (PLEG): container finished" podID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerID="a89da2c5e5156a32be57c5271bfa8c18348043d5fbfd745b2bf8478036372589" exitCode=0 Sep 10 09:44:47.932908 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.932906 2546 generic.go:356] "Generic (PLEG): container finished" podID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerID="1634933b86634a2c656c6de34e1ed2d103d906205c1ad0d5e35e17a6679b0b64" exitCode=0 Sep 10 09:44:47.933269 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.932949 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerDied","Data":"a89da2c5e5156a32be57c5271bfa8c18348043d5fbfd745b2bf8478036372589"} Sep 10 09:44:47.933269 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.932989 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerDied","Data":"1634933b86634a2c656c6de34e1ed2d103d906205c1ad0d5e35e17a6679b0b64"} Sep 10 09:44:47.935737 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.935710 2546 generic.go:356] "Generic (PLEG): container finished" podID="173b7933-7291-40d1-b33c-dc19066eef19" containerID="bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0" exitCode=0 Sep 10 09:44:47.935737 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.935730 2546 generic.go:356] "Generic (PLEG): container finished" podID="173b7933-7291-40d1-b33c-dc19066eef19" containerID="09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3" exitCode=0 Sep 10 09:44:47.935861 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.935748 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerDied","Data":"bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0"} Sep 10 09:44:47.935861 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:47.935774 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerDied","Data":"09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3"} Sep 10 09:44:51.692035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:51.691998 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 10 09:44:52.066233 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:52.066191 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 10 09:44:52.228122 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:52.228088 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:44:52.292216 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:52.292178 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 10 09:44:55.380235 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:44:55.380198 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 10 09:45:01.691590 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:01.691550 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 10 09:45:02.066927 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:02.066885 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 10 09:45:02.292520 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:02.292477 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 10 09:45:05.381045 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:05.380998 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 10 09:45:09.488299 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:09.488252 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:09.535733 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:09.535701 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:11.691062 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:11.691015 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 10 09:45:11.691620 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:11.691201 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:45:12.066895 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:12.066854 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 10 09:45:12.067077 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:12.066981 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:45:12.292663 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:12.292624 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 10 09:45:12.292836 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:12.292736 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:45:15.381167 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:15.381127 2546 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 10 09:45:15.381556 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:15.381264 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:45:17.048152 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.048121 2546 generic.go:356] "Generic (PLEG): container finished" podID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerID="ea777333dfb4f48c3cf40a41bb572fac5fc93c010e092ac81e7e4f7f6171d0e6" exitCode=137 Sep 10 09:45:17.048529 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.048194 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerDied","Data":"ea777333dfb4f48c3cf40a41bb572fac5fc93c010e092ac81e7e4f7f6171d0e6"} Sep 10 09:45:17.094874 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.094851 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:45:17.141480 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.141455 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-ui-tls\") pod \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " Sep 10 09:45:17.141589 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.141492 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4nr9\" (UniqueName: \"kubernetes.io/projected/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-kube-api-access-r4nr9\") pod \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " Sep 10 09:45:17.141589 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.141524 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-online-tls\") pod \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " Sep 10 09:45:17.141589 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.141547 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-offline-tls\") pod \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " Sep 10 09:45:17.141589 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.141566 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-registry-tls\") pod \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " Sep 10 09:45:17.141768 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.141603 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-feast-data\") pod \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\" (UID: \"2388b8d9-e68e-4b42-9d59-e8d8e69734c3\") " Sep 10 09:45:17.142133 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.142091 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2388b8d9-e68e-4b42-9d59-e8d8e69734c3" (UID: "2388b8d9-e68e-4b42-9d59-e8d8e69734c3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 09:45:17.142243 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.142206 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-feast-data" (OuterVolumeSpecName: "feast-data") pod "2388b8d9-e68e-4b42-9d59-e8d8e69734c3" (UID: "2388b8d9-e68e-4b42-9d59-e8d8e69734c3"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:45:17.143848 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.143825 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2388b8d9-e68e-4b42-9d59-e8d8e69734c3" (UID: "2388b8d9-e68e-4b42-9d59-e8d8e69734c3"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:45:17.144149 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.144129 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-online-tls" (OuterVolumeSpecName: "online-tls") pod "2388b8d9-e68e-4b42-9d59-e8d8e69734c3" (UID: "2388b8d9-e68e-4b42-9d59-e8d8e69734c3"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:45:17.144201 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.144143 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-kube-api-access-r4nr9" (OuterVolumeSpecName: "kube-api-access-r4nr9") pod "2388b8d9-e68e-4b42-9d59-e8d8e69734c3" (UID: "2388b8d9-e68e-4b42-9d59-e8d8e69734c3"). InnerVolumeSpecName "kube-api-access-r4nr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:45:17.144201 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.144150 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2388b8d9-e68e-4b42-9d59-e8d8e69734c3" (UID: "2388b8d9-e68e-4b42-9d59-e8d8e69734c3"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:45:17.242148 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.242123 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.242148 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.242147 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.242321 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.242159 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.242321 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.242169 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.242321 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.242180 2546 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-ui-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.242321 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.242191 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4nr9\" (UniqueName: \"kubernetes.io/projected/2388b8d9-e68e-4b42-9d59-e8d8e69734c3-kube-api-access-r4nr9\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.709495 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.709469 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:45:17.745554 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.745520 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-ui-tls\") pod \"173b7933-7291-40d1-b33c-dc19066eef19\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " Sep 10 09:45:17.745554 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.745551 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-online-tls\") pod \"173b7933-7291-40d1-b33c-dc19066eef19\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " Sep 10 09:45:17.745756 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.745580 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g4df\" (UniqueName: \"kubernetes.io/projected/173b7933-7291-40d1-b33c-dc19066eef19-kube-api-access-9g4df\") pod \"173b7933-7291-40d1-b33c-dc19066eef19\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " Sep 10 09:45:17.745756 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.745612 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-offline-tls\") pod \"173b7933-7291-40d1-b33c-dc19066eef19\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " Sep 10 09:45:17.745756 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.745654 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-registry-tls\") pod \"173b7933-7291-40d1-b33c-dc19066eef19\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " Sep 10 09:45:17.745756 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.745681 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/173b7933-7291-40d1-b33c-dc19066eef19-feast-data\") pod \"173b7933-7291-40d1-b33c-dc19066eef19\" (UID: \"173b7933-7291-40d1-b33c-dc19066eef19\") " Sep 10 09:45:17.746406 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.746377 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173b7933-7291-40d1-b33c-dc19066eef19-feast-data" (OuterVolumeSpecName: "feast-data") pod "173b7933-7291-40d1-b33c-dc19066eef19" (UID: "173b7933-7291-40d1-b33c-dc19066eef19"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:45:17.747832 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.747804 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "173b7933-7291-40d1-b33c-dc19066eef19" (UID: "173b7933-7291-40d1-b33c-dc19066eef19"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:45:17.748098 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.748071 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-online-tls" (OuterVolumeSpecName: "online-tls") pod "173b7933-7291-40d1-b33c-dc19066eef19" (UID: "173b7933-7291-40d1-b33c-dc19066eef19"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:45:17.748193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.748142 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173b7933-7291-40d1-b33c-dc19066eef19-kube-api-access-9g4df" (OuterVolumeSpecName: "kube-api-access-9g4df") pod "173b7933-7291-40d1-b33c-dc19066eef19" (UID: "173b7933-7291-40d1-b33c-dc19066eef19"). InnerVolumeSpecName "kube-api-access-9g4df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:45:17.748193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.748153 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "173b7933-7291-40d1-b33c-dc19066eef19" (UID: "173b7933-7291-40d1-b33c-dc19066eef19"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:45:17.748193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.748164 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "173b7933-7291-40d1-b33c-dc19066eef19" (UID: "173b7933-7291-40d1-b33c-dc19066eef19"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 09:45:17.846968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.846906 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g4df\" (UniqueName: \"kubernetes.io/projected/173b7933-7291-40d1-b33c-dc19066eef19-kube-api-access-9g4df\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.846968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.846931 2546 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-offline-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.846968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.846940 2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-registry-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.846968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.846949 2546 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/173b7933-7291-40d1-b33c-dc19066eef19-feast-data\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.846968 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.846957 2546 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-ui-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:17.847200 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:17.846982 2546 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/173b7933-7291-40d1-b33c-dc19066eef19-online-tls\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:18.054047 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.054016 2546 generic.go:356] "Generic (PLEG): container finished" podID="173b7933-7291-40d1-b33c-dc19066eef19" containerID="c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506" exitCode=137 Sep 10 09:45:18.054047 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.054042 2546 generic.go:356] "Generic (PLEG): container finished" podID="173b7933-7291-40d1-b33c-dc19066eef19" containerID="77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2" exitCode=137 Sep 10 09:45:18.054442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.054082 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" Sep 10 09:45:18.054442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.054099 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerDied","Data":"c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506"} Sep 10 09:45:18.054442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.054155 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerDied","Data":"77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2"} Sep 10 09:45:18.054442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.054175 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4" event={"ID":"173b7933-7291-40d1-b33c-dc19066eef19","Type":"ContainerDied","Data":"cad281ad8396123d8abc8f872d2fb9d41123995990fa04e73b538a99f5f04db1"} Sep 10 09:45:18.054442 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.054199 2546 scope.go:117] "RemoveContainer" containerID="bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0" Sep 10 09:45:18.056805 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.056779 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" event={"ID":"2388b8d9-e68e-4b42-9d59-e8d8e69734c3","Type":"ContainerDied","Data":"255edf4fd25806122ffbd299f5f9bb6ddce87b4ac82f374d38705296777629a4"} Sep 10 09:45:18.056914 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.056833 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt" Sep 10 09:45:18.063307 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.063272 2546 scope.go:117] "RemoveContainer" containerID="c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506" Sep 10 09:45:18.070031 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.070017 2546 scope.go:117] "RemoveContainer" containerID="09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3" Sep 10 09:45:18.074253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.074231 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt"] Sep 10 09:45:18.077169 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.077155 2546 scope.go:117] "RemoveContainer" containerID="77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2" Sep 10 09:45:18.079958 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.079940 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-8vtwt"] Sep 10 09:45:18.084245 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.084228 2546 scope.go:117] "RemoveContainer" containerID="897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4" Sep 10 09:45:18.089376 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.089357 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4"] Sep 10 09:45:18.091876 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.091853 2546 scope.go:117] "RemoveContainer" containerID="a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007" Sep 10 09:45:18.092700 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.092679 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-4v5g4"] Sep 10 09:45:18.098608 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.098591 2546 scope.go:117] "RemoveContainer" containerID="bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0" Sep 10 09:45:18.098870 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:18.098853 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0\": container with ID starting with bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0 not found: ID does not exist" containerID="bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0" Sep 10 09:45:18.098912 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.098876 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0"} err="failed to get container status \"bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0\": rpc error: code = NotFound desc = could not find container \"bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0\": container with ID starting with bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0 not found: ID does not exist" Sep 10 09:45:18.098912 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.098895 2546 scope.go:117] "RemoveContainer" containerID="c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506" Sep 10 09:45:18.099140 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:18.099122 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506\": container with ID starting with c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506 not found: ID does not exist" containerID="c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506" Sep 10 09:45:18.099193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.099145 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506"} err="failed to get container status \"c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506\": rpc error: code = NotFound desc = could not find container \"c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506\": container with ID starting with c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506 not found: ID does not exist" Sep 10 09:45:18.099193 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.099160 2546 scope.go:117] "RemoveContainer" containerID="09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3" Sep 10 09:45:18.099413 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:18.099394 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3\": container with ID starting with 09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3 not found: ID does not exist" containerID="09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3" Sep 10 09:45:18.099465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.099416 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3"} err="failed to get container status \"09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3\": rpc error: code = NotFound desc = could not find container \"09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3\": container with ID starting with 09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3 not found: ID does not exist" Sep 10 09:45:18.099465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.099430 2546 scope.go:117] "RemoveContainer" containerID="77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2" Sep 10 09:45:18.099685 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:18.099670 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2\": container with ID starting with 77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2 not found: ID does not exist" containerID="77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2" Sep 10 09:45:18.099743 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.099689 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2"} err="failed to get container status \"77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2\": rpc error: code = NotFound desc = could not find container \"77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2\": container with ID starting with 77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2 not found: ID does not exist" Sep 10 09:45:18.099743 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.099701 2546 scope.go:117] "RemoveContainer" containerID="897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4" Sep 10 09:45:18.099930 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:18.099910 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4\": container with ID starting with 897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4 not found: ID does not exist" containerID="897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4" Sep 10 09:45:18.099971 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.099934 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4"} err="failed to get container status \"897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4\": rpc error: code = NotFound desc = could not find container \"897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4\": container with ID starting with 897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4 not found: ID does not exist" Sep 10 09:45:18.099971 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.099948 2546 scope.go:117] "RemoveContainer" containerID="a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007" Sep 10 09:45:18.100173 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:18.100154 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007\": container with ID starting with a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007 not found: ID does not exist" containerID="a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007" Sep 10 09:45:18.100212 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.100177 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007"} err="failed to get container status \"a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007\": rpc error: code = NotFound desc = could not find container \"a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007\": container with ID starting with a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007 not found: ID does not exist" Sep 10 09:45:18.100212 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.100188 2546 scope.go:117] "RemoveContainer" containerID="bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0" Sep 10 09:45:18.100418 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.100400 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0"} err="failed to get container status \"bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0\": rpc error: code = NotFound desc = could not find container \"bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0\": container with ID starting with bc0c2086a61c6d1b39572872aafcc85cfeec5ff34185b9c60e342ccdae3a47d0 not found: ID does not exist" Sep 10 09:45:18.100474 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.100419 2546 scope.go:117] "RemoveContainer" containerID="c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506" Sep 10 09:45:18.100651 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.100632 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506"} err="failed to get container status \"c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506\": rpc error: code = NotFound desc = could not find container \"c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506\": container with ID starting with c7ec50ba34aa31c93cfb4d22f98311a2cc59ecc455a3ef61883b163039a9d506 not found: ID does not exist" Sep 10 09:45:18.100694 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.100652 2546 scope.go:117] "RemoveContainer" containerID="09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3" Sep 10 09:45:18.100862 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.100845 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3"} err="failed to get container status \"09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3\": rpc error: code = NotFound desc = could not find container \"09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3\": container with ID starting with 09117fde55b2f417a02a2cfd704c39516af29c66b1c77299809fa161d9ffb7c3 not found: ID does not exist" Sep 10 09:45:18.100862 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.100862 2546 scope.go:117] "RemoveContainer" containerID="77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2" Sep 10 09:45:18.101035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.101020 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2"} err="failed to get container status \"77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2\": rpc error: code = NotFound desc = could not find container \"77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2\": container with ID starting with 77d84b93f6c66e356b0040f1ad16e93f924f47e6d50c802b4c5196709055bbc2 not found: ID does not exist" Sep 10 09:45:18.101076 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.101035 2546 scope.go:117] "RemoveContainer" containerID="897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4" Sep 10 09:45:18.101240 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.101222 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4"} err="failed to get container status \"897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4\": rpc error: code = NotFound desc = could not find container \"897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4\": container with ID starting with 897cf4f528f57ca3f69dc01ff903b218d503c20487a695a2766efed4a7ca0df4 not found: ID does not exist" Sep 10 09:45:18.101240 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.101240 2546 scope.go:117] "RemoveContainer" containerID="a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007" Sep 10 09:45:18.101465 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.101450 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007"} err="failed to get container status \"a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007\": rpc error: code = NotFound desc = could not find container \"a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007\": container with ID starting with a35111500beb2f813b6e88e8102785fdc3e015bd77a430f20666ec794e9dd007 not found: ID does not exist" Sep 10 09:45:18.101504 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.101464 2546 scope.go:117] "RemoveContainer" containerID="a89da2c5e5156a32be57c5271bfa8c18348043d5fbfd745b2bf8478036372589" Sep 10 09:45:18.107752 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.107736 2546 scope.go:117] "RemoveContainer" containerID="ea777333dfb4f48c3cf40a41bb572fac5fc93c010e092ac81e7e4f7f6171d0e6" Sep 10 09:45:18.114538 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.114526 2546 scope.go:117] "RemoveContainer" containerID="1634933b86634a2c656c6de34e1ed2d103d906205c1ad0d5e35e17a6679b0b64" Sep 10 09:45:18.122676 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.122661 2546 scope.go:117] "RemoveContainer" containerID="58e5da6360fb5abf431e15b69965850b86f43ca9eb682c8302f153da1e053a8e" Sep 10 09:45:18.129045 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:18.129031 2546 scope.go:117] "RemoveContainer" containerID="9345466b0f26379a8f83718b0826bb9b5f4536bfdd6b4dc656366e7b6beab25a" Sep 10 09:45:19.260198 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:19.260160 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="173b7933-7291-40d1-b33c-dc19066eef19" path="/var/lib/kubelet/pods/173b7933-7291-40d1-b33c-dc19066eef19/volumes" Sep 10 09:45:19.260999 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:19.260976 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" path="/var/lib/kubelet/pods/2388b8d9-e68e-4b42-9d59-e8d8e69734c3/volumes" Sep 10 09:45:22.834496 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:22.834462 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:22.941251 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:22.941220 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:28.494434 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:28.494399 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:28.548867 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:28.548836 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:29.364932 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.364879 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rpg9p/must-gather-jhk47"] Sep 10 09:45:29.365140 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365128 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" Sep 10 09:45:29.365191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365141 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" Sep 10 09:45:29.365191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365150 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="registry" Sep 10 09:45:29.365191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365157 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="registry" Sep 10 09:45:29.365191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365165 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="feast-apply" Sep 10 09:45:29.365191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365171 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="feast-apply" Sep 10 09:45:29.365191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365178 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="feast-apply" Sep 10 09:45:29.365191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365183 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="feast-apply" Sep 10 09:45:29.365191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365190 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="feast-init" Sep 10 09:45:29.365191 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365196 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="feast-init" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365201 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="online" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365206 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="online" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365213 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="ui" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365232 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="ui" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365238 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="ui" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365244 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="ui" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365250 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="feast-init" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365255 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="feast-init" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365262 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="offline" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365267 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="offline" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365276 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="offline" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365295 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="offline" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365335 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="ui" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365342 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="online" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365348 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2388b8d9-e68e-4b42-9d59-e8d8e69734c3" containerName="offline" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365355 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="registry" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365360 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="offline" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365367 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="online" Sep 10 09:45:29.365509 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.365372 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="173b7933-7291-40d1-b33c-dc19066eef19" containerName="ui" Sep 10 09:45:29.368149 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.368130 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:29.370179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.370150 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rpg9p\"/\"openshift-service-ca.crt\"" Sep 10 09:45:29.370327 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.370264 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rpg9p\"/\"kube-root-ca.crt\"" Sep 10 09:45:29.370810 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.370792 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rpg9p\"/\"default-dockercfg-ppz9m\"" Sep 10 09:45:29.375950 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.375924 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpg9p/must-gather-jhk47"] Sep 10 09:45:29.386337 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.386270 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:29.435223 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.435198 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knd6\" (UniqueName: \"kubernetes.io/projected/8bdbeb48-fa5b-4872-b879-56b8cdae267a-kube-api-access-6knd6\") pod \"must-gather-jhk47\" (UID: \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\") " pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:29.435349 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.435305 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bdbeb48-fa5b-4872-b879-56b8cdae267a-must-gather-output\") pod \"must-gather-jhk47\" (UID: \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\") " pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:29.481531 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.481513 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:29.535556 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.535532 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bdbeb48-fa5b-4872-b879-56b8cdae267a-must-gather-output\") pod \"must-gather-jhk47\" (UID: \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\") " pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:29.535827 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.535575 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knd6\" (UniqueName: \"kubernetes.io/projected/8bdbeb48-fa5b-4872-b879-56b8cdae267a-kube-api-access-6knd6\") pod \"must-gather-jhk47\" (UID: \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\") " pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:29.535827 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.535809 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bdbeb48-fa5b-4872-b879-56b8cdae267a-must-gather-output\") pod \"must-gather-jhk47\" (UID: \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\") " pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:29.542522 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.542501 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knd6\" (UniqueName: \"kubernetes.io/projected/8bdbeb48-fa5b-4872-b879-56b8cdae267a-kube-api-access-6knd6\") pod \"must-gather-jhk47\" (UID: \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\") " pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:29.693675 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.693621 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:29.958966 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.958895 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rpg9p/must-gather-jhk47"] Sep 10 09:45:29.963261 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:45:29.963237 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bdbeb48_fa5b_4872_b879_56b8cdae267a.slice/crio-98aa672885a3574af68038e22188035b8b6e7196b827498d8357eb185306feb2 WatchSource:0}: Error finding container 98aa672885a3574af68038e22188035b8b6e7196b827498d8357eb185306feb2: Status 404 returned error can't find the container with id 98aa672885a3574af68038e22188035b8b6e7196b827498d8357eb185306feb2 Sep 10 09:45:29.965004 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:29.964986 2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 09:45:30.099398 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:30.099366 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpg9p/must-gather-jhk47" event={"ID":"8bdbeb48-fa5b-4872-b879-56b8cdae267a","Type":"ContainerStarted","Data":"98aa672885a3574af68038e22188035b8b6e7196b827498d8357eb185306feb2"} Sep 10 09:45:36.133029 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:36.132973 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpg9p/must-gather-jhk47" event={"ID":"8bdbeb48-fa5b-4872-b879-56b8cdae267a","Type":"ContainerStarted","Data":"b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f"} Sep 10 09:45:37.138168 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:37.138124 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpg9p/must-gather-jhk47" event={"ID":"8bdbeb48-fa5b-4872-b879-56b8cdae267a","Type":"ContainerStarted","Data":"a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16"} Sep 10 09:45:37.152963 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:37.152893 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rpg9p/must-gather-jhk47" podStartSLOduration=2.172720472 podStartE2EDuration="8.152873508s" totalImagesPullingTime="5.980153036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 09:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:45:37.152262884 +0000 UTC m=+1876.333951950" watchObservedRunningTime="2026-09-10 09:45:37.152873508 +0000 UTC m=+1876.334562575" Sep 10 09:45:46.172466 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:46.172424 2546 generic.go:356] "Generic (PLEG): container finished" podID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" containerID="b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f" exitCode=0 Sep 10 09:45:46.172854 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:46.172487 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rpg9p/must-gather-jhk47" event={"ID":"8bdbeb48-fa5b-4872-b879-56b8cdae267a","Type":"ContainerDied","Data":"b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f"} Sep 10 09:45:46.172854 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:46.172734 2546 scope.go:117] "RemoveContainer" containerID="b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f" Sep 10 09:45:46.202226 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:46.202202 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rpg9p_must-gather-jhk47_8bdbeb48-fa5b-4872-b879-56b8cdae267a/gather/0.log" Sep 10 09:45:46.522032 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:46.522003 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:46.659850 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:46.659813 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:46.722512 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:46.722488 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:49.373777 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:49.373741 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jr854_ce6a92df-68a8-49fa-acb8-3b40e6852595/global-pull-secret-syncer/0.log" Sep 10 09:45:49.419762 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:49.419738 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" Sep 10 09:45:49.531937 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:49.531914 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-170.ec2.internal_a131841e4a19fc7a8712e64d2e2b1c9d/haproxy/0.log" Sep 10 09:45:51.503947 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.503911 2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-rpg9p/must-gather-jhk47"] Sep 10 09:45:51.504459 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.504090 2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-rpg9p/must-gather-jhk47" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" containerName="copy" containerID="cri-o://a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16" gracePeriod=2 Sep 10 09:45:51.508450 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.507777 2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-rpg9p/must-gather-jhk47"] Sep 10 09:45:51.726525 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.726503 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rpg9p_must-gather-jhk47_8bdbeb48-fa5b-4872-b879-56b8cdae267a/copy/0.log" Sep 10 09:45:51.728289 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.728256 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:51.729998 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:51.729974 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jhk47\" is forbidden: User \"system:node:ip-10-0-139-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rpg9p\": no relationship found between node 'ip-10-0-139-170.ec2.internal' and this object" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:51.812002 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.811947 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6knd6\" (UniqueName: \"kubernetes.io/projected/8bdbeb48-fa5b-4872-b879-56b8cdae267a-kube-api-access-6knd6\") pod \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\" (UID: \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\") " Sep 10 09:45:51.812144 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.812129 2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bdbeb48-fa5b-4872-b879-56b8cdae267a-must-gather-output\") pod \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\" (UID: \"8bdbeb48-fa5b-4872-b879-56b8cdae267a\") " Sep 10 09:45:51.812479 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.812456 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdbeb48-fa5b-4872-b879-56b8cdae267a-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "8bdbeb48-fa5b-4872-b879-56b8cdae267a" (UID: "8bdbeb48-fa5b-4872-b879-56b8cdae267a"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 09:45:51.813906 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.813881 2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdbeb48-fa5b-4872-b879-56b8cdae267a-kube-api-access-6knd6" (OuterVolumeSpecName: "kube-api-access-6knd6") pod "8bdbeb48-fa5b-4872-b879-56b8cdae267a" (UID: "8bdbeb48-fa5b-4872-b879-56b8cdae267a"). InnerVolumeSpecName "kube-api-access-6knd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 09:45:51.913174 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.913148 2546 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8bdbeb48-fa5b-4872-b879-56b8cdae267a-must-gather-output\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:51.913174 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:51.913169 2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6knd6\" (UniqueName: \"kubernetes.io/projected/8bdbeb48-fa5b-4872-b879-56b8cdae267a-kube-api-access-6knd6\") on node \"ip-10-0-139-170.ec2.internal\" DevicePath \"\"" Sep 10 09:45:52.059694 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.059667 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:52.157420 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.157346 2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9qc8_57b8bb57-7b18-4390-94cc-7a8e8a0c811e/konnectivity-agent/0.log" limit=1024 Sep 10 09:45:52.194230 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.194208 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rpg9p_must-gather-jhk47_8bdbeb48-fa5b-4872-b879-56b8cdae267a/copy/0.log" Sep 10 09:45:52.194630 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.194607 2546 generic.go:356] "Generic (PLEG): container finished" podID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" containerID="a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16" exitCode=143 Sep 10 09:45:52.194698 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.194657 2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:52.194698 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.194672 2546 scope.go:117] "RemoveContainer" containerID="a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16" Sep 10 09:45:52.196576 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:52.196547 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jhk47\" is forbidden: User \"system:node:ip-10-0-139-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rpg9p\": no relationship found between node 'ip-10-0-139-170.ec2.internal' and this object" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:52.205310 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.205264 2546 scope.go:117] "RemoveContainer" containerID="b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f" Sep 10 09:45:52.207470 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:52.207447 2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jhk47\" is forbidden: User \"system:node:ip-10-0-139-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rpg9p\": no relationship found between node 'ip-10-0-139-170.ec2.internal' and this object" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" pod="openshift-must-gather-rpg9p/must-gather-jhk47" Sep 10 09:45:52.218401 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.218381 2546 scope.go:117] "RemoveContainer" containerID="a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16" Sep 10 09:45:52.218658 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:52.218637 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16\": container with ID starting with a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16 not found: ID does not exist" containerID="a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16" Sep 10 09:45:52.218704 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.218668 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16"} err="failed to get container status \"a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16\": rpc error: code = NotFound desc = could not find container \"a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16\": container with ID starting with a63c34eb540122fa0bb03fb9529971db607ff012146bade95e8091dbc60a4d16 not found: ID does not exist" Sep 10 09:45:52.218704 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.218685 2546 scope.go:117] "RemoveContainer" containerID="b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f" Sep 10 09:45:52.218951 ip-10-0-139-170 kubenswrapper[2546]: E0910 09:45:52.218936 2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f\": container with ID starting with b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f not found: ID does not exist" containerID="b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f" Sep 10 09:45:52.218995 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:52.218954 2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f"} err="failed to get container status \"b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f\": rpc error: code = NotFound desc = could not find container \"b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f\": container with ID starting with b6137b09ccba6b58dc8fa5bed829962fab4604c4cdfaa551757b3c13b72cf56f not found: ID does not exist" Sep 10 09:45:53.260119 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.260084 2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" path="/var/lib/kubelet/pods/8bdbeb48-fa5b-4872-b879-56b8cdae267a/volumes" Sep 10 09:45:53.422456 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.422424 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e2c8fc12-a867-4cde-b675-9499a9d3abf7/alertmanager/0.log" Sep 10 09:45:53.446794 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.446770 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e2c8fc12-a867-4cde-b675-9499a9d3abf7/config-reloader/0.log" Sep 10 09:45:53.472750 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.472728 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e2c8fc12-a867-4cde-b675-9499a9d3abf7/kube-rbac-proxy-web/0.log" Sep 10 09:45:53.492653 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.492636 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e2c8fc12-a867-4cde-b675-9499a9d3abf7/kube-rbac-proxy/0.log" Sep 10 09:45:53.511765 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.511711 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e2c8fc12-a867-4cde-b675-9499a9d3abf7/kube-rbac-proxy-metric/0.log" Sep 10 09:45:53.531157 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.531134 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e2c8fc12-a867-4cde-b675-9499a9d3abf7/prom-label-proxy/0.log" Sep 10 09:45:53.552092 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.552073 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e2c8fc12-a867-4cde-b675-9499a9d3abf7/init-config-reloader/0.log" Sep 10 09:45:53.632009 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.631992 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-cmstd_2754ca8d-cc92-4c02-9a54-89c57e442cba/kube-state-metrics/0.log" Sep 10 09:45:53.651497 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.651478 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-cmstd_2754ca8d-cc92-4c02-9a54-89c57e442cba/kube-rbac-proxy-main/0.log" Sep 10 09:45:53.673917 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.673900 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-cmstd_2754ca8d-cc92-4c02-9a54-89c57e442cba/kube-rbac-proxy-self/0.log" Sep 10 09:45:53.703789 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.703771 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-55d65cd66d-ckfrn_e97928d4-5581-4a1a-a354-a1a8e712bded/metrics-server/0.log" Sep 10 09:45:53.728596 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.728576 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-d9266_6dc4c77a-7daf-4279-b807-221620251b0a/monitoring-plugin/0.log" Sep 10 09:45:53.900144 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.900076 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slhtd_8c1e873e-f584-47d4-9f40-8282069fd755/node-exporter/0.log" Sep 10 09:45:53.920153 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.920128 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slhtd_8c1e873e-f584-47d4-9f40-8282069fd755/kube-rbac-proxy/0.log" Sep 10 09:45:53.944225 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:53.944192 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slhtd_8c1e873e-f584-47d4-9f40-8282069fd755/init-textfile/0.log" Sep 10 09:45:54.240104 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:54.240067 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-vq98v_a7f1db2c-9409-44bd-8439-6810c5952810/prometheus-operator-admission-webhook/0.log" Sep 10 09:45:54.280252 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:54.280232 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bcdcf84cb-9zrsh_1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c/telemeter-client/0.log" Sep 10 09:45:54.298179 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:54.298155 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bcdcf84cb-9zrsh_1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c/reload/0.log" Sep 10 09:45:54.317533 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:54.317513 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-bcdcf84cb-9zrsh_1c1b6821-4f2b-44ab-9f51-bf7abe4b6c3c/kube-rbac-proxy/0.log" Sep 10 09:45:56.494641 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:56.494611 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64dfb6bb58-9qbgh_f4c646f4-ea2f-4622-aab5-97004536e447/console/0.log" Sep 10 09:45:57.149630 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.149595 2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg"] Sep 10 09:45:57.149896 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.149884 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" containerName="gather" Sep 10 09:45:57.149940 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.149898 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" containerName="gather" Sep 10 09:45:57.149940 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.149912 2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" containerName="copy" Sep 10 09:45:57.149940 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.149918 2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" containerName="copy" Sep 10 09:45:57.150035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.149958 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" containerName="gather" Sep 10 09:45:57.150035 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.149967 2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bdbeb48-fa5b-4872-b879-56b8cdae267a" containerName="copy" Sep 10 09:45:57.151905 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.151890 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.153837 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.153810 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mnrtq\"/\"kube-root-ca.crt\"" Sep 10 09:45:57.153973 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.153823 2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mnrtq\"/\"openshift-service-ca.crt\"" Sep 10 09:45:57.154480 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.154458 2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mnrtq\"/\"default-dockercfg-7dxl6\"" Sep 10 09:45:57.159654 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.159635 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg"] Sep 10 09:45:57.255232 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.255203 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-proc\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.255384 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.255269 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-lib-modules\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.255384 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.255343 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-sys\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.255384 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.255373 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6q5\" (UniqueName: \"kubernetes.io/projected/55fd5c52-ad68-4493-9769-36b3ad21a2df-kube-api-access-cr6q5\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.255556 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.255406 2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-podres\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.356429 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.356383 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-proc\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.356429 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.356436 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-lib-modules\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.356666 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.356489 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-sys\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.356666 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.356498 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-proc\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.356666 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.356510 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6q5\" (UniqueName: \"kubernetes.io/projected/55fd5c52-ad68-4493-9769-36b3ad21a2df-kube-api-access-cr6q5\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.356666 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.356564 2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-podres\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.356842 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.356699 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-lib-modules\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.356842 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.356717 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-sys\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.356842 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.356701 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55fd5c52-ad68-4493-9769-36b3ad21a2df-podres\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.363790 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.363769 2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6q5\" (UniqueName: \"kubernetes.io/projected/55fd5c52-ad68-4493-9769-36b3ad21a2df-kube-api-access-cr6q5\") pod \"perf-node-gather-daemonset-lg5lg\" (UID: \"55fd5c52-ad68-4493-9769-36b3ad21a2df\") " pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.462385 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.462318 2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:57.726292 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.726246 2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg"] Sep 10 09:45:57.728757 ip-10-0-139-170 kubenswrapper[2546]: W0910 09:45:57.728720 2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod55fd5c52_ad68_4493_9769_36b3ad21a2df.slice/crio-ae7e29d5124daa7bc71563a5e3401da8fad4fecc8b257ca60774eb79b4051e4f WatchSource:0}: Error finding container ae7e29d5124daa7bc71563a5e3401da8fad4fecc8b257ca60774eb79b4051e4f: Status 404 returned error can't find the container with id ae7e29d5124daa7bc71563a5e3401da8fad4fecc8b257ca60774eb79b4051e4f Sep 10 09:45:57.750146 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.750128 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c2m2f_7961c4df-e598-459c-b080-e369916c1c64/dns/0.log" Sep 10 09:45:57.771090 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.771064 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c2m2f_7961c4df-e598-459c-b080-e369916c1c64/kube-rbac-proxy/0.log" Sep 10 09:45:57.891330 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:57.891305 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mjg8s_0ed4f23e-1ad2-4346-8550-93565896543d/dns-node-resolver/0.log" Sep 10 09:45:58.218637 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:58.218606 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" event={"ID":"55fd5c52-ad68-4493-9769-36b3ad21a2df","Type":"ContainerStarted","Data":"17d06e786a6ffd29ba76101e1335ea530a4445708d548267e193ead8eff189b0"} Sep 10 09:45:58.218763 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:58.218659 2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" event={"ID":"55fd5c52-ad68-4493-9769-36b3ad21a2df","Type":"ContainerStarted","Data":"ae7e29d5124daa7bc71563a5e3401da8fad4fecc8b257ca60774eb79b4051e4f"} Sep 10 09:45:58.218763 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:58.218751 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:45:58.234936 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:58.234882 2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" podStartSLOduration=1.234865398 podStartE2EDuration="1.234865398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 09:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 09:45:58.233603308 +0000 UTC m=+1897.415292397" watchObservedRunningTime="2026-09-10 09:45:58.234865398 +0000 UTC m=+1897.416554462" Sep 10 09:45:58.321491 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:58.321460 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79d77b6f48-zkjn8_5101d125-9003-4e52-8c8d-be6a11fabdbd/registry/0.log" Sep 10 09:45:58.341946 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:58.341924 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-htq4b_8615e2e1-f518-4237-960f-2eda0b516165/node-ca/0.log" Sep 10 09:45:59.343087 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:59.343061 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xd4qq_34992b33-8b5a-4f69-9f8a-bc88b37f340a/serve-healthcheck-canary/0.log" Sep 10 09:45:59.805116 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:59.805084 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zfhl5_bad5d836-bb05-41eb-afc6-1fde320819f5/kube-rbac-proxy/0.log" Sep 10 09:45:59.822660 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:59.822639 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zfhl5_bad5d836-bb05-41eb-afc6-1fde320819f5/exporter/0.log" Sep 10 09:45:59.840633 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:45:59.840611 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zfhl5_bad5d836-bb05-41eb-afc6-1fde320819f5/extractor/0.log" Sep 10 09:46:04.232599 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:04.232566 2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mnrtq/perf-node-gather-daemonset-lg5lg" Sep 10 09:46:05.004780 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.004753 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5tt2m_bcdf75da-e1be-4a9d-a514-fbd4e04e4766/kube-multus-additional-cni-plugins/0.log" Sep 10 09:46:05.023781 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.023756 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5tt2m_bcdf75da-e1be-4a9d-a514-fbd4e04e4766/egress-router-binary-copy/0.log" Sep 10 09:46:05.043695 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.043666 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5tt2m_bcdf75da-e1be-4a9d-a514-fbd4e04e4766/cni-plugins/0.log" Sep 10 09:46:05.061623 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.061600 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5tt2m_bcdf75da-e1be-4a9d-a514-fbd4e04e4766/bond-cni-plugin/0.log" Sep 10 09:46:05.081949 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.081932 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5tt2m_bcdf75da-e1be-4a9d-a514-fbd4e04e4766/routeoverride-cni/0.log" Sep 10 09:46:05.101189 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.101165 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5tt2m_bcdf75da-e1be-4a9d-a514-fbd4e04e4766/whereabouts-cni-bincopy/0.log" Sep 10 09:46:05.120429 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.120409 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5tt2m_bcdf75da-e1be-4a9d-a514-fbd4e04e4766/whereabouts-cni/0.log" Sep 10 09:46:05.366134 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.366049 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zwqd9_3f56cc9f-529f-4a84-a940-a9bef6f86110/kube-multus/0.log" Sep 10 09:46:05.470670 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.470637 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-knj5j_ea243c6d-7906-4f8f-8da3-5e1d888d4d6a/network-metrics-daemon/0.log" Sep 10 09:46:05.485992 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:05.485967 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-knj5j_ea243c6d-7906-4f8f-8da3-5e1d888d4d6a/kube-rbac-proxy/0.log" Sep 10 09:46:06.702217 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:06.702185 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xsql9_b8589e0d-ca1b-4278-9f30-1979024023f2/ovn-controller/0.log" Sep 10 09:46:06.716253 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:06.716229 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xsql9_b8589e0d-ca1b-4278-9f30-1979024023f2/ovn-acl-logging/0.log" Sep 10 09:46:06.739271 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:06.739248 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xsql9_b8589e0d-ca1b-4278-9f30-1979024023f2/kube-rbac-proxy-node/0.log" Sep 10 09:46:06.757803 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:06.757777 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xsql9_b8589e0d-ca1b-4278-9f30-1979024023f2/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 09:46:06.773878 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:06.773826 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xsql9_b8589e0d-ca1b-4278-9f30-1979024023f2/northd/0.log" Sep 10 09:46:06.791835 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:06.791816 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xsql9_b8589e0d-ca1b-4278-9f30-1979024023f2/nbdb/0.log" Sep 10 09:46:06.813947 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:06.813925 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xsql9_b8589e0d-ca1b-4278-9f30-1979024023f2/sbdb/0.log" Sep 10 09:46:06.859701 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:06.859677 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xsql9_b8589e0d-ca1b-4278-9f30-1979024023f2/ovnkube-controller/0.log" Sep 10 09:46:07.804051 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:07.803972 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-b49hb_52423870-4667-4de2-b4ee-6418b9edefaf/network-check-target-container/0.log" Sep 10 09:46:08.681746 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:08.681719 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-ctr4v_0edb23e0-a253-42a0-a1dd-6f17199ba610/iptables-alerter/0.log" Sep 10 09:46:09.229834 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:09.229806 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-dwbqv_afb5107c-5103-4a50-8f07-b9325b1aba7d/tuned/0.log" Sep 10 09:46:12.367893 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:12.367863 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wtt9j_64f62386-59d2-4fe5-994a-0f4c61c3a0ca/csi-driver/0.log" Sep 10 09:46:12.384554 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:12.384523 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wtt9j_64f62386-59d2-4fe5-994a-0f4c61c3a0ca/csi-node-driver-registrar/0.log" Sep 10 09:46:12.401247 ip-10-0-139-170 kubenswrapper[2546]: I0910 09:46:12.401226 2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wtt9j_64f62386-59d2-4fe5-994a-0f4c61c3a0ca/csi-liveness-probe/0.log"